2018-01-13 07:25:13,689 INFO [main] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458, deleteOnExit=true 2018-01-13 07:25:13,693 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cache_data in system properties and HBase conf 2018-01-13 07:25:13,693 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/hadoop_tmp in system properties and HBase conf 2018-01-13 07:25:13,693 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/hadoop_logs in system properties and HBase conf 2018-01-13 07:25:13,694 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/mapred_local in system properties and HBase conf 2018-01-13 07:25:13,694 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/mapred_temp in system properties and HBase conf 2018-01-13 07:25:13,695 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2018-01-13 07:25:13,808 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-01-13 07:25:14,227 DEBUG [main] fs.HFileSystem(318): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-01-13 07:25:15,615 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-01-13 07:25:15,847 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-01-13 07:25:15,919 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 07:25:15,950 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_50754_hdfs____.k0q8cb/webapp 2018-01-13 07:25:16,138 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50754 2018-01-13 07:25:16,953 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 07:25:16,959 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_46053_datanode____.munf2l/webapp 2018-01-13 07:25:17,089 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46053 2018-01-13 07:25:17,394 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 07:25:17,401 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_40097_datanode____.p1mdvr/webapp 2018-01-13 07:25:17,535 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40097 2018-01-13 07:25:18,378 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:45316] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-13 07:25:18,378 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:45316] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-13 07:25:18,471 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1370a2ed78af47: from storage DS-5ba0c480-698e-4fa2-be8d-55b114c682ac node DatanodeRegistration(127.0.0.1:59189, datanodeUuid=e361b17c-883b-4f18-a1bd-92fb43f01ad1, infoPort=36700, infoSecurePort=0, ipcPort=47899, storageInfo=lv=-56;cid=testClusterID;nsid=809542926;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs 2018-01-13 07:25:18,473 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1370a2ed78b21a: from storage DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189 node DatanodeRegistration(127.0.0.1:49352, datanodeUuid=f9f33522-cf14-4d6d-9f8d-50a8a8519fd9, infoPort=40917, infoSecurePort=0, ipcPort=58968, storageInfo=lv=-56;cid=testClusterID;nsid=809542926;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-01-13 07:25:18,473 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1370a2ed78af47: from storage DS-c2f91fd0-9011-4287-bd13-96644a3511e6 node DatanodeRegistration(127.0.0.1:59189, datanodeUuid=e361b17c-883b-4f18-a1bd-92fb43f01ad1, infoPort=36700, infoSecurePort=0, ipcPort=47899, storageInfo=lv=-56;cid=testClusterID;nsid=809542926;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-13 07:25:18,473 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1370a2ed78b21a: from storage DS-c601089d-c719-49d5-8041-8d8fdc76b7cc node DatanodeRegistration(127.0.0.1:49352, datanodeUuid=f9f33522-cf14-4d6d-9f8d-50a8a8519fd9, infoPort=40917, infoSecurePort=0, ipcPort=58968, storageInfo=lv=-56;cid=testClusterID;nsid=809542926;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-13 07:25:19,329 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-13 07:25:19,346 INFO [main] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=59094 2018-01-13 07:25:19,384 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testCreateWithParents Thread=105, OpenFileDescriptor=684, MaxFileDescriptor=60000, SystemLoadAverage=218, ProcessCount=271, AvailableMemoryMB=11361 2018-01-13 07:25:19,405 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:19,410 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:19,802 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 7 2018-01-13 07:25:19,803 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741825_1001 size 7 2018-01-13 07:25:20,219 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:25:20,220 INFO [main] hbase.HBaseTestingUtility(1204): Setting hbase.fs.tmp.dir to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:25:20,440 INFO [main] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-01-13 07:25:20,683 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:20,705 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:20,707 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:20,707 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:20,835 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:21,016 DEBUG [main] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-01-13 07:25:21,106 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:39126 2018-01-13 07:25:21,122 INFO [main] hfile.CacheConfig(553): Allocating On heap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-01-13 07:25:21,131 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,131 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,134 DEBUG [main] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-01-13 07:25:21,136 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,138 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,192 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:39126 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:21,491 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:391260x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:21,493 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:39126-0x160ee6894500000 connected 2018-01-13 07:25:21,592 DEBUG [main] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:21,594 DEBUG [main] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:21,621 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=39126 2018-01-13 07:25:21,623 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=39126 2018-01-13 07:25:21,625 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=39126 2018-01-13 07:25:21,631 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:21,656 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:21,656 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:21,656 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:21,657 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:21,657 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:21,659 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:49200 2018-01-13 07:25:21,660 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,660 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,661 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,664 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,667 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:49200 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:21,674 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:492000x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:21,675 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:49200-0x160ee6894500001 connected 2018-01-13 07:25:21,682 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:25:21,691 DEBUG [main] zookeeper.ZKUtil(357): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:21,693 DEBUG [main] zookeeper.ZKUtil(357): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:21,693 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=49200 2018-01-13 07:25:21,694 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=49200 2018-01-13 07:25:21,695 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=49200 2018-01-13 07:25:21,696 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:21,763 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:21,763 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:21,763 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:21,764 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:21,768 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:21,769 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:21,772 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:54666 2018-01-13 07:25:21,773 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,773 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,776 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,780 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,784 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:54666 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:21,890 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:546660x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:21,891 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:54666-0x160ee6894500002 connected 2018-01-13 07:25:21,891 DEBUG [main] zookeeper.ZKUtil(357): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:21,892 DEBUG [main] zookeeper.ZKUtil(357): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:21,893 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=54666 2018-01-13 07:25:21,894 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=54666 2018-01-13 07:25:21,895 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=54666 2018-01-13 07:25:21,921 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:21,922 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:21,922 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:21,922 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:21,923 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:21,923 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:21,926 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:41843 2018-01-13 07:25:21,926 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,927 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:21,929 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,934 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:21,936 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:41843 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:21,948 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:418430x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:21,949 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:41843-0x160ee6894500003 connected 2018-01-13 07:25:21,950 DEBUG [main] zookeeper.ZKUtil(357): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:21,951 DEBUG [main] zookeeper.ZKUtil(357): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:21,951 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=41843 2018-01-13 07:25:21,953 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=41843 2018-01-13 07:25:21,954 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=41843 2018-01-13 07:25:21,955 INFO [M:0;asf911:39126] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:21,955 INFO [M:1;asf911:49200] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,49200,1515828321654 2018-01-13 07:25:21,974 DEBUG [M:1;asf911:49200] zookeeper.ZKUtil(355): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,49200,1515828321654 2018-01-13 07:25:21,974 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(355): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:22,016 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:22,016 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:22,016 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:22,016 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:22,016 DEBUG [M:1;asf911:49200] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:25:22,021 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(355): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:22,023 DEBUG [M:1;asf911:49200] zookeeper.ZKUtil(355): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:22,024 INFO [M:0;asf911:39126] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,39126,1515828320287 from backup master directory 2018-01-13 07:25:22,028 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:22,028 INFO [M:1;asf911:49200] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,39126,1515828320287; waiting to become the next active master 2018-01-13 07:25:22,029 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:22,029 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:22,028 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:22,040 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:22,041 WARN [M:0;asf911:39126] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:22,041 INFO [M:0;asf911:39126] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:22,046 INFO [M:0;asf911:39126] regionserver.ChunkCreator(408): Allocating MemStoreChunkPool with chunk size 2 MB, max count 497, initial count 0 2018-01-13 07:25:22,176 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:22,176 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:22,180 DEBUG [M:0;asf911:39126] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 33e31897-a326-4882-9524-81747e322a8a 2018-01-13 07:25:22,226 INFO [M:0;asf911:39126] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:25:22,234 INFO [M:0;asf911:39126] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:25:22,279 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:22,284 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741827_1003 size 32 2018-01-13 07:25:22,289 DEBUG [M:0;asf911:39126] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:22,341 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:22,360 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:22,364 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:22,375 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:22,397 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:22,397 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:22,398 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:22,399 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:22,404 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:22,405 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:22,405 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:22,406 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:22,412 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:22,412 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:22,413 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:22,414 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:22,420 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:22,420 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:22,421 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:22,421 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:22,435 DEBUG [M:0;asf911:39126] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:22,454 DEBUG [M:0;asf911:39126] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:22,462 DEBUG [M:0;asf911:39126] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:22,463 INFO [M:0;asf911:39126] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:25:22,463 DEBUG [M:0;asf911:39126] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:22,464 DEBUG [M:0;asf911:39126] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:22,465 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:22,466 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:22,466 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:22,466 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:22,467 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:22,467 INFO [M:0;asf911:39126] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:22,509 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:22,511 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:22,517 DEBUG [M:0;asf911:39126] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:22,559 INFO [M:0;asf911:39126] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:22,580 INFO [M:0;asf911:39126] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:25:22,581 DEBUG [M:0;asf911:39126] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:25:22,637 INFO [M:0;asf911:39126] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x160e7bc4 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:22,692 DEBUG [M:0;asf911:39126] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a0ce55e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:22,732 INFO [M:0;asf911:39126] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:25:22,736 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:25:22,737 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:25:22,750 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:25:22,750 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:25:22,782 INFO [M:0;asf911:39126] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:25:22,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:22,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:22,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:22,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:22,808 INFO [M:0;asf911:39126] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,39126,1515828320287, sessionid=0x160ee6894500000, setting cluster-up flag (Was=false) 2018-01-13 07:25:22,873 INFO [M:0;asf911:39126] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:25:22,876 DEBUG [M:0;asf911:39126] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:22,923 INFO [M:0;asf911:39126] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:25:22,925 DEBUG [M:0;asf911:39126] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:22,961 INFO [RS:0;asf911:54666] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x51c3ac07 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:22,961 INFO [RS:1;asf911:41843] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x220bb261 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:22,984 DEBUG [RS:0;asf911:54666] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f04484f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:22,985 DEBUG [RS:1;asf911:41843] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ccc4850, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:22,986 INFO [RS:0;asf911:54666] regionserver.HRegionServer(866): ClusterId : 33e31897-a326-4882-9524-81747e322a8a 2018-01-13 07:25:22,987 INFO [RS:1;asf911:41843] regionserver.HRegionServer(866): ClusterId : 33e31897-a326-4882-9524-81747e322a8a 2018-01-13 07:25:22,990 INFO [M:0;asf911:39126] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:25:22,992 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:39126, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:22,992 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:39126, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:22,992 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:39126, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:22,992 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:39126, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:22,993 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:39126, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:22,993 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:22,994 DEBUG [M:0;asf911:39126] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:39126, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:22,993 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:23,006 DEBUG [RS:1;asf911:41843] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:23,015 DEBUG [RS:0;asf911:54666] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:23,017 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:23,017 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:23,017 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:23,017 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:23,023 DEBUG [RS:0;asf911:54666] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:23,023 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:25:23,026 INFO [M:0;asf911:39126] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:25:23,031 DEBUG [RS:1;asf911:41843] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:23,033 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:23,033 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:23,033 DEBUG [RS:0;asf911:54666] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b3bc777, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:23,034 DEBUG [RS:1;asf911:41843] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28f3d911, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:23,038 DEBUG [RS:0;asf911:54666] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:54666 2018-01-13 07:25:23,039 DEBUG [M:0;asf911:39126] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:25:23,038 DEBUG [RS:1;asf911:41843] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:41843 2018-01-13 07:25:23,041 INFO [M:0;asf911:39126] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:25:23,041 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 15msec 2018-01-13 07:25:23,043 DEBUG [M:0;asf911:39126] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:25:23,043 DEBUG [M:0;asf911:39126] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:25:23,044 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 2msec 2018-01-13 07:25:23,044 INFO [RS:0;asf911:54666] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:23,046 INFO [RS:0;asf911:54666] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:23,044 INFO [RS:1;asf911:41843] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:23,046 INFO [RS:1;asf911:41843] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:23,049 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828353048 2018-01-13 07:25:23,050 INFO [M:0;asf911:39126] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:25:23,051 INFO [RS:1;asf911:41843] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,39126,1515828320287 with port=41843, startcode=1515828321918 2018-01-13 07:25:23,051 INFO [RS:0;asf911:54666] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,39126,1515828320287 with port=54666, startcode=1515828321760 2018-01-13 07:25:23,054 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:25:23,056 INFO [M:0;asf911:39126] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:23,057 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:25:23,058 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:25:23,058 INFO [M:0;asf911:39126] cleaner.CleanerChore(113): Cleaner pool size is 8 2018-01-13 07:25:23,058 INFO [M:0;asf911:39126] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:25:23,063 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:25:23,065 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:25:23,065 DEBUG [M:0;asf911:39126-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:23,066 DEBUG [M:0;asf911:39126-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500007 connected 2018-01-13 07:25:23,066 DEBUG [M:0;asf911:39126] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:25:23,069 DEBUG [M:0;asf911:39126] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:39126-HFileCleaner.large.0-1515828323068,5,main] 2018-01-13 07:25:23,069 DEBUG [M:0;asf911:39126] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:39126-HFileCleaner.small.0-1515828323069,5,main] 2018-01-13 07:25:23,080 INFO [M:0;asf911:39126] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828322007, completionTime=-1 2018-01-13 07:25:23,080 INFO [M:0;asf911:39126] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:25:23,101 DEBUG [RS:1;asf911:41843] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:39126 2018-01-13 07:25:23,101 DEBUG [RS:0;asf911:54666] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:39126 2018-01-13 07:25:23,173 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.1 (auth:SIMPLE) 2018-01-13 07:25:23,173 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.0 (auth:SIMPLE) 2018-01-13 07:25:23,183 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 58943 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:23,183 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 37759 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:23,229 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39126] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,230 INFO [M:0;asf911:39126] master.ServerManager(871): Waiting on RegionServer count=1 to settle; waited=150ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:25:23,230 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39126] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,254 DEBUG [RS:1;asf911:41843] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:23,254 DEBUG [RS:0;asf911:54666] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:23,254 DEBUG [RS:1;asf911:41843] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:23,255 DEBUG [RS:1;asf911:41843] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:23,255 DEBUG [RS:0;asf911:54666] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:23,256 DEBUG [RS:0;asf911:54666] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:23,281 INFO [M:0;asf911:39126] master.ServerManager(891): Finished wait on RegionServer count=2; waited=201ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:25:23,299 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:23,300 DEBUG [RS:1;asf911:41843] zookeeper.ZKUtil(355): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,300 DEBUG [RS:0;asf911:54666] zookeeper.ZKUtil(355): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,300 WARN [RS:1;asf911:41843] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:23,300 WARN [RS:0;asf911:54666] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:23,301 DEBUG [RS:0;asf911:54666] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,302 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,301 DEBUG [RS:1;asf911:41843] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,305 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,313 INFO [M:0;asf911:39126] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:25:23,327 INFO [RS:0;asf911:54666] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:23,327 INFO [RS:1;asf911:41843] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:23,335 DEBUG [RS:0;asf911:54666] zookeeper.ZKUtil(355): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,335 DEBUG [RS:1;asf911:41843] zookeeper.ZKUtil(355): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,336 DEBUG [RS:0;asf911:54666] zookeeper.ZKUtil(355): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,336 DEBUG [RS:1;asf911:41843] zookeeper.ZKUtil(355): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,343 INFO [RS:1;asf911:41843] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x488d1093 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:23,343 INFO [RS:0;asf911:54666] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1dc0f3bb to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:23,362 DEBUG [RS:1;asf911:41843] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@525c09f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:23,362 DEBUG [RS:0;asf911:54666] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@521ef91f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:23,362 DEBUG [RS:1;asf911:41843] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:23,363 DEBUG [RS:0;asf911:54666] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:23,375 INFO [RS:0;asf911:54666] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:23,375 INFO [RS:1;asf911:41843] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:23,412 INFO [RS:1;asf911:41843] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:23,412 INFO [RS:0;asf911:54666] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:23,421 INFO [RS:0;asf911:54666] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:23,421 INFO [RS:1;asf911:41843] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:23,424 INFO [RS:0;asf911:54666] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:23,424 INFO [RS:1;asf911:41843] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:23,446 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:41843, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,446 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:54666, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,446 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:41843, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:23,447 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:54666, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:23,447 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:41843, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,447 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:54666, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,447 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:41843, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,447 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:54666, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,447 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:41843, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:23,448 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:54666, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:23,448 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:41843, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:23,448 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:54666, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:23,448 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:41843, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:23,448 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:54666, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:23,449 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:41843, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,449 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:54666, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:23,449 DEBUG [RS:1;asf911:41843] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:41843, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:23,449 DEBUG [RS:0;asf911:54666] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:54666, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:23,493 INFO [SplitLogWorker-asf911:41843] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,41843,1515828321918 starting 2018-01-13 07:25:23,494 INFO [SplitLogWorker-asf911:54666] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,54666,1515828321760 starting 2018-01-13 07:25:23,499 INFO [RS:0;asf911:54666] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:23,499 INFO [RS:1;asf911:41843] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:23,554 INFO [RS:0;asf911:54666] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,54666,1515828321760, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:54666, sessionid=0x160ee6894500002 2018-01-13 07:25:23,555 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,41843,1515828321918, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:41843, sessionid=0x160ee6894500003 2018-01-13 07:25:23,555 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:23,555 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:23,555 DEBUG [RS:1;asf911:41843] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,555 DEBUG [RS:0;asf911:54666] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,556 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,41843,1515828321918' 2018-01-13 07:25:23,556 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,54666,1515828321760' 2018-01-13 07:25:23,557 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:23,557 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:23,558 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:23,558 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:23,559 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:23,559 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:23,559 DEBUG [M:0;asf911:39126] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:23,559 DEBUG [RS:0;asf911:54666] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:23,560 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,54666,1515828321760' 2018-01-13 07:25:23,560 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:23,559 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:23,560 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:23,560 DEBUG [RS:1;asf911:41843] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:23,561 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,41843,1515828321918' 2018-01-13 07:25:23,561 DEBUG [RS:0;asf911:54666] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:23,561 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:23,562 DEBUG [RS:0;asf911:54666] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:23,562 DEBUG [RS:1;asf911:41843] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:23,562 INFO [RS:0;asf911:54666] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:23,562 INFO [RS:0;asf911:54666] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:23,563 DEBUG [RS:1;asf911:41843] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:23,563 INFO [RS:1;asf911:41843] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:23,563 INFO [RS:1;asf911:41843] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:23,569 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:23,570 INFO [ProcExecWrkr-2] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:23,573 DEBUG [ProcExecWrkr-2] zookeeper.ZKUtil(614): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:23,578 DEBUG [ProcExecWrkr-2] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:25:23,630 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:23,631 DEBUG [ProcExecWrkr-2] zookeeper.ZKUtil(614): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:23,631 INFO [ProcExecWrkr-2] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:25:23,637 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:25:23,698 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:25:23,702 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:25:23,857 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:25:23,866 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:25:23,886 INFO [ProcExecWrkr-4] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:24,589 INFO [RS:0;asf911:54666] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C54666%2C1515828321760, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,54666,1515828321760, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:24,589 INFO [RS:1;asf911:41843] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C41843%2C1515828321918, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:24,625 DEBUG [RS:1;asf911:41843] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(417): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-01-13 07:25:24,675 DEBUG [RS:1;asf911:41843] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(490): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:488) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:552) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:628) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:504) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:69) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:44) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:56) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:263) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2119) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1314) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1195) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:25:24,679 DEBUG [RS:1;asf911:41843] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(524): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-01-13 07:25:24,710 DEBUG [RS-EventLoopGroup-5-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:657) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:117) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:716) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:711) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:25:24,714 DEBUG [RS-EventLoopGroup-5-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:25:24,714 DEBUG [RS-EventLoopGroup-5-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:24,714 DEBUG [RS-EventLoopGroup-5-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:24,717 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:24,763 INFO [RS:0;asf911:54666] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,54666,1515828321760/asf911.gq1.ygridcore.net%2C54666%2C1515828321760.1515828324613 2018-01-13 07:25:24,763 INFO [RS:1;asf911:41843] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918/asf911.gq1.ygridcore.net%2C41843%2C1515828321918.1515828324613 2018-01-13 07:25:24,763 DEBUG [RS:0;asf911:54666] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:25:24,763 DEBUG [RS:1;asf911:41843] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:25:24,794 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:24,839 DEBUG [ProcExecWrkr-5] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:25:24,858 INFO [ProcExecWrkr-5] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,010 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,41843,1515828321918 version=3145728 2018-01-13 07:25:25,057 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,061 DEBUG [RSProcedureDispatcher-pool3-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:25,068 INFO [RS-EventLoopGroup-5-13] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:25,068 INFO [RS-EventLoopGroup-5-13] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49968 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:25,076 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=41843] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:25:25,081 INFO [RS_OPEN_META-asf911:41843-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:25,090 INFO [RS_OPEN_META-asf911:41843-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C41843%2C1515828321918.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:25,107 DEBUG [RS-EventLoopGroup-5-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:25,107 DEBUG [RS-EventLoopGroup-5-15] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:25,117 INFO [RS_OPEN_META-asf911:41843-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918/asf911.gq1.ygridcore.net%2C41843%2C1515828321918.meta.1515828325093.meta 2018-01-13 07:25:25,117 DEBUG [RS_OPEN_META-asf911:41843-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:25:25,118 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:25,148 DEBUG [RS_OPEN_META-asf911:41843-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:25:25,168 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:25:25,178 INFO [RS_OPEN_META-asf911:41843-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:25:25,186 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:25:25,186 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:25,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:25,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:25,198 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:25,199 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:25,199 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:25,204 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:25,204 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:25,204 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:25,204 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:25,205 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:25,209 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:25,209 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:25,210 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:25,210 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:25,211 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:25,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:25,215 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:25,215 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:25,216 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:25,216 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:25,220 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:25,220 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:25,221 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:25,221 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:25,222 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:25,229 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:25,233 DEBUG [RS_OPEN_META-asf911:41843-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:25,243 DEBUG [RS_OPEN_META-asf911:41843-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:25:25,244 INFO [RS_OPEN_META-asf911:41843-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:25:25,295 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:25:25,316 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39126] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,321 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:25:25,325 DEBUG [RS_OPEN_META-asf911:41843-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,394 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828325330}]},"ts":1515828325330} 2018-01-13 07:25:25,459 DEBUG [hconnection-0xea916b7-shared-pool7-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:25,472 INFO [RS-EventLoopGroup-5-19] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:25,473 INFO [RS-EventLoopGroup-5-19] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49971 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:25,539 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:25:25,540 INFO [ProcExecWrkr-7] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:25,611 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:25:25,665 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:25:25,666 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.9210sec 2018-01-13 07:25:25,824 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 2.3510sec 2018-01-13 07:25:25,825 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39126] assignment.AssignmentManager(996): META REPORTED: rit=OPEN, location=asf911.gq1.ygridcore.net,41843,1515828321918, table=hbase:meta, region=1588230740 2018-01-13 07:25:25,826 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39126] assignment.AssignmentManager(998): META REPORTED but no procedure found (complete?); set location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,826 INFO [M:0;asf911:39126] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:25,917 INFO [M:0;asf911:39126] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,41843,1515828321918 belongs to an existing region server 2018-01-13 07:25:25,919 INFO [M:0;asf911:39126] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,54666,1515828321760 belongs to an existing region server 2018-01-13 07:25:25,919 INFO [M:0;asf911:39126] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:25:25,926 INFO [M:0;asf911:39126] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:25:25,927 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828330927 2018-01-13 07:25:25,927 INFO [M:0;asf911:39126] assignment.AssignmentManager(1219): Joined the cluster in 8msec, failover=true 2018-01-13 07:25:25,969 INFO [M:0;asf911:39126] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:25:25,976 INFO [M:0;asf911:39126] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:25:26,142 DEBUG [M:0;asf911:39126] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:26,144 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:26,199 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:26,234 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:26,234 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:26,239 DEBUG [ProcExecWrkr-8] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:26,243 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:25:26,270 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:26,271 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:26,273 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,274 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d.: disabling compactions & flushes 2018-01-13 07:25:26,274 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,274 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,361 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:26,373 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:25:26,387 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:25:26,441 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:26,441 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828326441}]},"ts":1515828326441} 2018-01-13 07:25:26,450 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:25:26,451 DEBUG [ProcExecWrkr-8] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,465 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918}] 2018-01-13 07:25:26,517 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918 hbase:namespace hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,525 INFO [ProcExecWrkr-9] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918; rit=OFFLINE, location=asf911.gq1.ygridcore.net,41843,1515828321918; forceNewPlan=false, retain=false 2018-01-13 07:25:26,690 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:25:26,692 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:26,695 INFO [ProcExecWrkr-10] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1ffb223c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:26,717 DEBUG [ProcExecWrkr-10] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bc4b51b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:26,720 DEBUG [MultiHConnection-shared--pool8-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:26,733 INFO [RS-EventLoopGroup-5-21] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:26,734 INFO [RS-EventLoopGroup-5-21] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49982 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:26,745 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918; rit=OPENING, location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:26,896 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,41843,1515828321918 version=3145728 2018-01-13 07:25:26,899 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=41843] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,909 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] regionserver.HRegion(6949): Opening region: {ENCODED => cc7583ca664aa45ff9132791a2e9344d, NAME => 'hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:26,911 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace cc7583ca664aa45ff9132791a2e9344d 2018-01-13 07:25:26,911 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,919 DEBUG [StoreOpener-cc7583ca664aa45ff9132791a2e9344d-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/cc7583ca664aa45ff9132791a2e9344d/info 2018-01-13 07:25:26,919 DEBUG [StoreOpener-cc7583ca664aa45ff9132791a2e9344d-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/cc7583ca664aa45ff9132791a2e9344d/info 2018-01-13 07:25:26,920 INFO [StoreOpener-cc7583ca664aa45ff9132791a2e9344d-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:26,920 INFO [StoreOpener-cc7583ca664aa45ff9132791a2e9344d-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:26,921 INFO [StoreOpener-cc7583ca664aa45ff9132791a2e9344d-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:26,925 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/cc7583ca664aa45ff9132791a2e9344d 2018-01-13 07:25:26,936 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/cc7583ca664aa45ff9132791a2e9344d/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:26,936 INFO [RS_OPEN_PRIORITY_REGION-asf911:41843-0] regionserver.HRegion(960): Onlined cc7583ca664aa45ff9132791a2e9344d; next sequenceid=2 2018-01-13 07:25:26,939 INFO [PostOpenDeployTasks:cc7583ca664aa45ff9132791a2e9344d] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,943 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39126] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918; rit=OPENING, location=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:26,945 INFO [ProcExecWrkr-11] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:26,946 DEBUG [PostOpenDeployTasks:cc7583ca664aa45ff9132791a2e9344d] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:26,947 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:41843-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. on asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:27,008 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:27,052 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:25:27,052 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:27,053 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=cc7583ca664aa45ff9132791a2e9344d, target=asf911.gq1.ygridcore.net,41843,1515828321918 in 492msec 2018-01-13 07:25:27,053 DEBUG [ProcExecWrkr-12] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828327053}]},"ts":1515828327053} 2018-01-13 07:25:27,059 INFO [ProcExecWrkr-12] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:25:27,074 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:25:27,081 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:27,128 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:27,176 DEBUG [M:0;asf911:39126] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,225 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1520sec 2018-01-13 07:25:27,226 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,286 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,344 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,419 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,431 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:27,494 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:27,552 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 390msec 2018-01-13 07:25:27,720 DEBUG [M:0;asf911:39126] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:27,722 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:27,782 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:27,840 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:27,908 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:27,923 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:27,983 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:28,041 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 418msec 2018-01-13 07:25:28,065 DEBUG [M:0;asf911:39126] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:25:28,073 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:28,081 DEBUG [M:0;asf911:39126] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:25:28,090 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:28,090 INFO [M:0;asf911:39126] master.HMaster(928): Master has completed initialization 6.048sec 2018-01-13 07:25:28,096 INFO [M:0;asf911:39126] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:25:28,096 INFO [M:0;asf911:39126] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:25:28,104 INFO [M:0;asf911:39126] regionserver.HRegionServer(866): ClusterId : 33e31897-a326-4882-9524-81747e322a8a 2018-01-13 07:25:28,168 INFO [main] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1a3e8e24 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:28,184 DEBUG [main] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ed4a7e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:28,205 DEBUG [hconnection-0x51a18b21-shared-pool12-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:28,221 INFO [RS-EventLoopGroup-5-22] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:28,222 INFO [RS-EventLoopGroup-5-22] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49994 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:28,228 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:28,229 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=org.apache.hadoop.hbase.TestZooKeeper connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:28,240 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:28,241 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c connected 2018-01-13 07:25:28,317 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2 2018-01-13 07:25:28,319 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2/l3 2018-01-13 07:25:28,321 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2/l3/l4 2018-01-13 07:25:28,323 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2/l3/l4/testCreateWithParents 2018-01-13 07:25:28,340 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2/l3/l4/testCreateWithParents 2018-01-13 07:25:28,340 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1/l2/l3/l4 2018-01-13 07:25:28,340 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2/l3/l4 2018-01-13 07:25:28,340 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1/l2/l3 2018-01-13 07:25:28,341 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2/l3 2018-01-13 07:25:28,341 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1/l2 2018-01-13 07:25:28,341 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2 2018-01-13 07:25:28,341 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1 2018-01-13 07:25:28,341 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1 2018-01-13 07:25:28,357 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450000c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/testCreateWithParents 2018-01-13 07:25:28,362 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,54666,1515828321760: killAll ***** 2018-01-13 07:25:28,363 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:28,374 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1209532416, "init" : 792723456, "max" : 2609905664, "used" : 484657368 }, "NonHeapMemoryUsage" : { "committed" : 105316352, "init" : 2555904, "max" : -1, "used" : 103856736 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ] } 2018-01-13 07:25:28,384 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39126] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,54666,1515828321760 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,54666,1515828321760: killAll ***** 2018-01-13 07:25:28,389 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,54666,1515828321760' ***** 2018-01-13 07:25:28,390 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:28,390 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,41843,1515828321918: killAll ***** 2018-01-13 07:25:28,390 INFO [RS:0;asf911:54666] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:28,390 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:28,391 INFO [RS:0;asf911:54666] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:28,391 INFO [SplitLogWorker-asf911:54666] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:28,391 INFO [SplitLogWorker-asf911:54666] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,54666,1515828321760 exiting 2018-01-13 07:25:28,392 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:28,392 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:28,392 INFO [RS:0;asf911:54666] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:28,393 INFO [RS:0;asf911:54666] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:28,393 INFO [RS:0;asf911:54666] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:28,393 DEBUG [RS:0;asf911:54666] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:28,394 INFO [RS:0;asf911:54666] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x51c3ac07 to localhost:59094 2018-01-13 07:25:28,394 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1209532416, "init" : 792723456, "max" : 2609905664, "used" : 485906240 }, "NonHeapMemoryUsage" : { "committed" : 105578496, "init" : 2555904, "max" : -1, "used" : 104008736 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net" } ] } 2018-01-13 07:25:28,394 DEBUG [RS:0;asf911:54666] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,395 INFO [RS:0;asf911:54666] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,54666,1515828321760; all regions closed. 2018-01-13 07:25:28,396 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39126] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,41843,1515828321918 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,41843,1515828321918: killAll ***** 2018-01-13 07:25:28,398 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,41843,1515828321918' ***** 2018-01-13 07:25:28,398 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:28,398 INFO [RS:1;asf911:41843] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:28,398 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:28,399 INFO [SplitLogWorker-asf911:41843] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:28,400 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:28,400 INFO [RS:1;asf911:41843] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:28,400 INFO [SplitLogWorker-asf911:41843] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,41843,1515828321918 exiting 2018-01-13 07:25:28,401 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:28,401 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,39126,1515828320287' ***** 2018-01-13 07:25:28,401 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:28,401 INFO [RS:1;asf911:41843] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:28,402 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:28,403 INFO [RS:1;asf911:41843] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:28,404 DEBUG [M:0;asf911:39126] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b853239, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:28,404 INFO [M:0;asf911:39126] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:28,404 DEBUG [M:0;asf911:39126] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:28,405 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:28,406 DEBUG [RS:1;asf911:41843] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:28,407 DEBUG [RS_CLOSE_REGION-asf911:41843-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:28,407 INFO [RS:1;asf911:41843] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x220bb261 to localhost:59094 2018-01-13 07:25:28,409 DEBUG [RS:1;asf911:41843] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,411 DEBUG [RS_CLOSE_REGION-asf911:41843-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d.: disabling compactions & flushes 2018-01-13 07:25:28,412 INFO [M:0;asf911:39126] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x160e7bc4 to localhost:59094 2018-01-13 07:25:28,412 DEBUG [RS_CLOSE_REGION-asf911:41843-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:28,413 INFO [StoreCloserThread-hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d.-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:28,412 INFO [RS:1;asf911:41843] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:28,414 ERROR [RS_CLOSE_REGION-asf911:41843-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:25:28,416 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:28,413 DEBUG [M:0;asf911:39126] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,416 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:28,414 INFO [RS:1;asf911:41843] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:28,416 INFO [M:0;asf911:39126] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,39126,1515828320287; all regions closed. 2018-01-13 07:25:28,416 INFO [RS:1;asf911:41843] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:28,417 DEBUG [M:0;asf911:39126] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,417 INFO [M:0;asf911:39126] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,39126,1515828320287 had [[ScheduledChore: Name: CatalogJanitor-asf911:39126 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,39126,1515828320287-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,39126,1515828320287-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,39126,1515828320287-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,39126,1515828320287-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,39126,1515828320287-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS]] on shutdown 2018-01-13 07:25:28,417 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:25:28,419 DEBUG [RS:1;asf911:41843] regionserver.HRegionServer(1392): Online Regions={cc7583ca664aa45ff9132791a2e9344d=hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d., 1588230740=hbase:meta,,1.1588230740} 2018-01-13 07:25:28,417 INFO [RS_CLOSE_REGION-asf911:41843-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:28,419 INFO [M:0;asf911:39126] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:25:28,421 INFO [M:0;asf911:39126] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:25:28,418 DEBUG [RS_CLOSE_META-asf911:41843-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:25:28,421 DEBUG [M:0;asf911:39126] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:28,421 DEBUG [RS_CLOSE_REGION-asf911:41843-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828325970.cc7583ca664aa45ff9132791a2e9344d. 2018-01-13 07:25:28,423 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:28,423 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:28,424 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:28,422 DEBUG [RS_CLOSE_META-asf911:41843-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:28,426 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,49200,1515828321654' ***** 2018-01-13 07:25:28,426 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:28,423 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:28,423 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:28,426 DEBUG [RS_CLOSE_META-asf911:41843-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:28,428 INFO [M:1;asf911:49200] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x0fb02c74 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:28,429 DEBUG [RS:0;asf911:54666] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,429 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:28,430 INFO [RS:0;asf911:54666] regionserver.Leases(147): RS:0;asf911:54666 closing leases 2018-01-13 07:25:28,430 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:28,430 INFO [RS:0;asf911:54666] regionserver.Leases(150): RS:0;asf911:54666 closed leases 2018-01-13 07:25:28,430 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:28,431 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:28,431 INFO [RS:0;asf911:54666] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,54666,1515828321760 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,54666,1515828321760 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:28,431 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:28,433 INFO [RS:0;asf911:54666] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:28,434 DEBUG [main] zookeeper.ZKUtil(614): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:28,434 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:28,433 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:28,432 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:28,432 DEBUG [M:0;asf911:39126] zookeeper.ZKUtil(614): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:28,435 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:28,435 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:28,435 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:28,436 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:25:28,434 ERROR [RS_CLOSE_META-asf911:41843-0] regionserver.HRegion(1620): Memstore size is 1330 2018-01-13 07:25:28,434 INFO [RS:0;asf911:54666] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:28,436 DEBUG [RS_CLOSE_META-asf911:41843-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:28,436 WARN [M:0;asf911:39126] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:28,436 INFO [RS:0;asf911:54666] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:28,437 INFO [M:0;asf911:39126] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:25:28,437 INFO [M:0;asf911:39126] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x1ffb223c to localhost:59094 2018-01-13 07:25:28,437 DEBUG [M:0;asf911:39126] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,438 INFO [RS_CLOSE_META-asf911:41843-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:28,439 DEBUG [RS_CLOSE_META-asf911:41843-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:28,438 INFO [RS:0;asf911:54666] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:28,439 INFO [M:0;asf911:39126] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:25:28,440 DEBUG [main] zookeeper.ZKUtil(614): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:28,441 INFO [M:0;asf911:39126] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:25:28,441 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:28,442 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,443 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,442 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,444 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,442 INFO [M:0;asf911:39126] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:25:28,446 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,446 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,445 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,445 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,444 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,444 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,444 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,444 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,452 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:28,453 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:28,443 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,443 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,443 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,443 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,39126,1515828320287 2018-01-13 07:25:28,454 DEBUG [M:1;asf911:49200] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e63026e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:28,452 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:28,459 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:28,446 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:28,465 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:28,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:28,465 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:28,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:28,475 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:28,476 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:28,476 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:28,475 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,54666,1515828321760 2018-01-13 07:25:28,477 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,54666,1515828321760] 2018-01-13 07:25:28,477 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741829_1005 size 12154 2018-01-13 07:25:28,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:54666-0x160ee6894500002, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:28,478 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; asf911.gq1.ygridcore.net,54666,1515828321760 expired; onlineServers=1 2018-01-13 07:25:28,479 INFO [M:0;asf911:39126] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,39126,1515828320287_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:28,479 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:39126-0x160ee6894500000, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:28,479 INFO [M:0;asf911:39126] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:25:28,479 INFO [M:0;asf911:39126] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:28,481 INFO [RS:0;asf911:54666] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,54666,1515828321760; zookeeper connection closed. 2018-01-13 07:25:28,482 INFO [RS:0;asf911:54666] regionserver.HRegionServer(1157): RS:0;asf911:54666 exiting 2018-01-13 07:25:28,483 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5daab543] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5daab543 2018-01-13 07:25:28,490 DEBUG [M:0;asf911:39126] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,39126,1515828320287 already deleted, retry=false 2018-01-13 07:25:28,491 INFO [asf911.gq1.ygridcore.net,41843,1515828321918_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 07:25:28,498 INFO [M:0;asf911:39126] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,39126,1515828320287; zookeeper connection closed. 2018-01-13 07:25:28,499 INFO [M:0;asf911:39126] regionserver.HRegionServer(1157): M:0;asf911:39126 exiting 2018-01-13 07:25:28,619 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,41843,1515828321918; all regions closed. 2018-01-13 07:25:28,625 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:28,626 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:28,634 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:28,635 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:28,636 DEBUG [RS:1;asf911:41843] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:28,638 INFO [RS:1;asf911:41843] regionserver.Leases(147): RS:1;asf911:41843 closing leases 2018-01-13 07:25:28,638 INFO [RS:1;asf911:41843] regionserver.Leases(150): RS:1;asf911:41843 closed leases 2018-01-13 07:25:28,639 INFO [RS:1;asf911:41843] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,41843,1515828321918 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,41843,1515828321918 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:28,639 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:28,639 INFO [RS:1;asf911:41843] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:28,649 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:41843-0x160ee6894500003, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,41843,1515828321918 2018-01-13 07:25:28,656 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,41843,1515828321918; zookeeper connection closed. 2018-01-13 07:25:28,658 INFO [RS:1;asf911:41843] regionserver.HRegionServer(1157): RS:1;asf911:41843 exiting 2018-01-13 07:25:28,658 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2511229e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2511229e 2018-01-13 07:25:28,658 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:25:29,461 INFO [M:1;asf911:49200] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:29,463 INFO [M:1;asf911:49200] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,49200,1515828321654 2018-01-13 07:25:29,464 DEBUG [M:1;asf911:49200] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:29,464 INFO [M:1;asf911:49200] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x0fb02c74 to localhost:59094 2018-01-13 07:25:29,464 DEBUG [M:1;asf911:49200] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:29,464 INFO [M:1;asf911:49200] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,49200,1515828321654; all regions closed. 2018-01-13 07:25:29,464 INFO [M:1;asf911:49200] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,49200,1515828321654 had [] on shutdown 2018-01-13 07:25:29,464 DEBUG [M:1;asf911:49200] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:29,474 DEBUG [M:1;asf911:49200] zookeeper.ZKUtil(614): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:29,474 WARN [M:1;asf911:49200] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:29,481 DEBUG [M:1;asf911:49200] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,49200,1515828321654 already deleted, retry=false 2018-01-13 07:25:29,490 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:49200-0x160ee6894500001, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,49200,1515828321654 2018-01-13 07:25:29,490 INFO [M:1;asf911:49200] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,49200,1515828321654; zookeeper connection closed. 2018-01-13 07:25:29,490 INFO [M:1;asf911:49200] regionserver.HRegionServer(1157): M:1;asf911:49200 exiting 2018-01-13 07:25:29,492 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,497 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741829_1005 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,497 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741830_1006 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,498 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741832_1008 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:29,498 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741831_1007 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:29,498 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741828_1004 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,498 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,499 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741833_1009 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:29,499 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741834_1010 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,499 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741826_1002 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:29,500 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741825_1001 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:29,501 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:29,515 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:29,516 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee689450000e connected 2018-01-13 07:25:29,517 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:25:29,518 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:25:29,518 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:25:29,519 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:25:29,520 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:25:29,520 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:25:29,521 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:25:29,522 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:25:29,522 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:25:29,523 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:25:29,524 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:25:29,525 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:25:29,525 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:25:29,529 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:25:29,530 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:25:29,531 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:25:29,533 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:25:29,534 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:25:29,534 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:25:29,537 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:25:29,538 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:25:29,579 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:29,579 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:25:29,580 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:25:29,581 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:25:29,581 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:29,581 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:25:29,581 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:29,583 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450000e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:25:29,622 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testCreateWithParents Thread=175 (was 105) Potentially hanging thread: main-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: ReadOnlyZKClient sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$62/2026761827.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: M:0;asf911:39126-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: ReadOnlyZKClient sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$62/2026761827.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: threadDeathWatcher-6-1 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:39126-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-5-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcClient-timer-pool1-t1 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: OldWALsCleaner-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:148) org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$0(LogCleaner.java:126) org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$71/1983510821.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:39126-HFileCleaner.small.0-1515828323069 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: ReadOnlyZKClient-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2062426709_1 at /127.0.0.1:54366 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-586658239_1 at /127.0.0.1:40243 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: OldWALsCleaner-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:148) org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$0(LogCleaner.java:126) org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$71/1983510821.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:39126-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: main-SendThread(localhost:59094) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool2-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-5-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) Potentially hanging thread: RS-EventLoopGroup-5-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: main-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: M:0;asf911:39126-HFileCleaner.large.0-1515828323068 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-5-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:307) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$62/2026761827.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@2ea9460c java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-5-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=1239 (was 684) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=208 (was 218), ProcessCount=270 (was 271), AvailableMemoryMB=11148 (was 11361) 2018-01-13 07:25:29,623 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1239 is superior to 1024 2018-01-13 07:25:29,648 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testCreateSilentIsReallySilent Thread=174, OpenFileDescriptor=1239, MaxFileDescriptor=60000, SystemLoadAverage=208, ProcessCount=270, AvailableMemoryMB=11149 2018-01-13 07:25:29,648 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1239 is superior to 1024 2018-01-13 07:25:29,669 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:29,670 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:29,674 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:25:29,674 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:25:29,679 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:29,679 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:29,680 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:29,680 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:29,681 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:29,683 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:59777 2018-01-13 07:25:29,683 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,684 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,685 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,688 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,690 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:59777 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:29,707 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:597770x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:29,708 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:59777-0x160ee689450000f connected 2018-01-13 07:25:29,774 DEBUG [main] zookeeper.ZKUtil(357): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:29,775 DEBUG [main] zookeeper.ZKUtil(357): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:29,775 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=59777 2018-01-13 07:25:29,777 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=59777 2018-01-13 07:25:29,778 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=59777 2018-01-13 07:25:29,778 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:29,781 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:29,781 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:29,782 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:29,782 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:29,782 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:29,784 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:46318 2018-01-13 07:25:29,785 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,785 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,786 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,789 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,791 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:46318 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:29,798 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:463180x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:29,799 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:46318-0x160ee6894500010 connected 2018-01-13 07:25:29,806 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:25:29,811 DEBUG [main] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:29,811 DEBUG [main] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:29,812 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=46318 2018-01-13 07:25:29,813 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=46318 2018-01-13 07:25:29,814 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=46318 2018-01-13 07:25:29,814 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:29,845 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:29,845 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:29,846 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:29,846 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:29,846 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:29,846 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:29,849 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:34337 2018-01-13 07:25:29,849 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,850 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,852 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,854 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,857 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:34337 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:29,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:343370x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:29,866 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:34337-0x160ee6894500011 connected 2018-01-13 07:25:29,867 DEBUG [main] zookeeper.ZKUtil(357): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:29,868 DEBUG [main] zookeeper.ZKUtil(357): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:29,868 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=34337 2018-01-13 07:25:29,869 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=34337 2018-01-13 07:25:29,870 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=34337 2018-01-13 07:25:29,912 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:29,912 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:29,913 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:29,913 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:29,913 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:29,914 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:29,918 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:57040 2018-01-13 07:25:29,919 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,919 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:29,921 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,922 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:29,925 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:57040 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:29,932 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:570400x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:29,933 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:57040-0x160ee6894500012 connected 2018-01-13 07:25:29,933 DEBUG [main] zookeeper.ZKUtil(357): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:29,934 DEBUG [main] zookeeper.ZKUtil(357): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:29,934 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=57040 2018-01-13 07:25:29,935 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=57040 2018-01-13 07:25:29,936 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=57040 2018-01-13 07:25:29,939 INFO [M:0;asf911:59777] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,59777,1515828329677 2018-01-13 07:25:29,939 INFO [M:1;asf911:46318] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:29,949 DEBUG [M:0;asf911:59777] zookeeper.ZKUtil(355): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,59777,1515828329677 2018-01-13 07:25:29,949 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(355): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:29,957 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:29,957 DEBUG [M:0;asf911:59777] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:25:29,957 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:29,957 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:29,957 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:29,958 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(355): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:29,961 DEBUG [M:0;asf911:59777] zookeeper.ZKUtil(355): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:29,963 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:29,963 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:29,963 INFO [M:0;asf911:59777] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,46318,1515828329779; waiting to become the next active master 2018-01-13 07:25:29,963 INFO [M:1;asf911:46318] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,46318,1515828329779 from backup master directory 2018-01-13 07:25:29,965 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:29,966 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:29,973 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:29,973 WARN [M:1;asf911:46318] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:29,973 INFO [M:1;asf911:46318] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:30,005 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:30,006 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:30,009 DEBUG [M:1;asf911:46318] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: b696352d-1ffd-427e-9e42-770be24a4b2e 2018-01-13 07:25:30,016 INFO [M:1;asf911:46318] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:25:30,017 INFO [M:1;asf911:46318] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:25:30,034 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:30,034 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:30,036 DEBUG [M:1;asf911:46318] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:30,043 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:30,044 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:30,044 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:30,044 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:30,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:30,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:30,049 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:30,049 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:30,053 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:30,054 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:30,054 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:30,054 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:30,059 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:30,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:30,059 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:30,060 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:30,065 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:30,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:30,066 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:30,068 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:30,071 DEBUG [M:1;asf911:46318] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:30,075 DEBUG [M:1;asf911:46318] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:30,080 DEBUG [M:1;asf911:46318] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:30,080 INFO [M:1;asf911:46318] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:25:30,080 DEBUG [M:1;asf911:46318] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:30,080 DEBUG [M:1;asf911:46318] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:30,081 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:30,082 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:30,082 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:30,082 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:30,082 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:30,082 INFO [M:1;asf911:46318] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:30,098 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:30,098 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:30,105 DEBUG [M:1;asf911:46318] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:30,127 INFO [M:1;asf911:46318] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:30,127 INFO [M:1;asf911:46318] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:25:30,128 DEBUG [M:1;asf911:46318] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:25:30,141 INFO [M:1;asf911:46318] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x78432573 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:30,158 DEBUG [M:1;asf911:46318] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3311efdf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:30,182 INFO [M:1;asf911:46318] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:25:30,183 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:25:30,184 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:25:30,193 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:25:30,197 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:25:30,200 INFO [M:1;asf911:46318] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:25:30,206 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:30,207 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:30,207 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:30,207 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:30,209 INFO [M:1;asf911:46318] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,46318,1515828329779, sessionid=0x160ee6894500010, setting cluster-up flag (Was=false) 2018-01-13 07:25:30,248 INFO [M:1;asf911:46318] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:25:30,250 DEBUG [M:1;asf911:46318] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:30,290 INFO [M:1;asf911:46318] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:25:30,292 DEBUG [M:1;asf911:46318] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:30,294 INFO [M:1;asf911:46318] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:25:30,294 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:46318, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:30,294 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:46318, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:30,294 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:46318, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:30,295 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:46318, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:30,295 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:46318, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:30,295 DEBUG [M:1;asf911:46318] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:46318, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:30,298 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:25:30,298 INFO [M:1;asf911:46318] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:25:30,302 DEBUG [M:1;asf911:46318] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:25:30,303 INFO [M:1;asf911:46318] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:25:30,303 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 5msec 2018-01-13 07:25:30,304 DEBUG [M:1;asf911:46318] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:25:30,304 DEBUG [M:1;asf911:46318] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:25:30,304 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 1msec 2018-01-13 07:25:30,308 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828360308 2018-01-13 07:25:30,308 INFO [M:1;asf911:46318] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:25:30,308 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:25:30,309 INFO [M:1;asf911:46318] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:30,310 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:25:30,310 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:25:30,311 INFO [M:1;asf911:46318] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:25:30,312 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:25:30,313 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:25:30,313 DEBUG [M:1;asf911:46318] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:25:30,313 DEBUG [M:1;asf911:46318] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:46318-HFileCleaner.large.0-1515828330313,5,main] 2018-01-13 07:25:30,314 DEBUG [M:1;asf911:46318] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:46318-HFileCleaner.small.0-1515828330313,5,main] 2018-01-13 07:25:30,314 INFO [M:1;asf911:46318] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828329949, completionTime=-1 2018-01-13 07:25:30,314 INFO [M:1;asf911:46318] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:25:30,315 DEBUG [M:1;asf911:46318-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:30,316 DEBUG [M:1;asf911:46318-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500014 connected 2018-01-13 07:25:30,344 INFO [RS:0;asf911:34337] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5f401d09 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:30,344 INFO [RS:1;asf911:57040] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x01853a41 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:30,381 DEBUG [RS:0;asf911:34337] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5187f891, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:30,383 DEBUG [RS:1;asf911:57040] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f42c401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:30,384 INFO [RS:0;asf911:34337] regionserver.HRegionServer(866): ClusterId : b696352d-1ffd-427e-9e42-770be24a4b2e 2018-01-13 07:25:30,384 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:30,390 DEBUG [RS:0;asf911:34337] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:30,390 INFO [RS:1;asf911:57040] regionserver.HRegionServer(866): ClusterId : b696352d-1ffd-427e-9e42-770be24a4b2e 2018-01-13 07:25:30,390 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:30,398 DEBUG [RS:1;asf911:57040] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:30,398 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:30,398 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:30,406 DEBUG [RS:0;asf911:34337] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:30,407 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:30,407 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:30,415 DEBUG [RS:1;asf911:57040] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:30,415 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:30,415 DEBUG [RS:0;asf911:34337] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55b87679, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:30,416 DEBUG [RS:0;asf911:34337] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:34337 2018-01-13 07:25:30,416 INFO [RS:0;asf911:34337] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:30,416 INFO [RS:0;asf911:34337] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:30,416 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:30,417 INFO [RS:0;asf911:34337] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,46318,1515828329779 with port=34337, startcode=1515828329843 2018-01-13 07:25:30,417 DEBUG [RS:1;asf911:57040] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7fce3a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:30,418 DEBUG [RS:1;asf911:57040] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:57040 2018-01-13 07:25:30,418 INFO [RS:1;asf911:57040] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:30,418 INFO [RS:1;asf911:57040] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:30,418 DEBUG [RS:0;asf911:34337] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:46318 2018-01-13 07:25:30,419 INFO [RS:1;asf911:57040] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,46318,1515828329779 with port=57040, startcode=1515828329910 2018-01-13 07:25:30,422 DEBUG [RS:1;asf911:57040] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:46318 2018-01-13 07:25:30,433 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.2 (auth:SIMPLE) 2018-01-13 07:25:30,434 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.3 (auth:SIMPLE) 2018-01-13 07:25:30,434 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 59547 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:30,434 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 57871 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:30,437 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46318] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,438 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46318] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,439 DEBUG [RS:1;asf911:57040] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:30,439 DEBUG [RS:0;asf911:34337] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:30,442 DEBUG [RS:0;asf911:34337] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:30,442 DEBUG [RS:0;asf911:34337] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:30,441 DEBUG [RS:1;asf911:57040] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:30,442 DEBUG [RS:1;asf911:57040] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:30,448 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:30,457 DEBUG [RS:0;asf911:34337] zookeeper.ZKUtil(355): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,457 WARN [RS:0;asf911:34337] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:30,457 DEBUG [RS:0;asf911:34337] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,457 DEBUG [RS:1;asf911:57040] zookeeper.ZKUtil(355): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,457 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,457 WARN [RS:1;asf911:57040] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:30,458 DEBUG [RS:1;asf911:57040] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,458 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,464 INFO [M:1;asf911:46318] master.ServerManager(891): Finished wait on RegionServer count=2; waited=150ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:25:30,466 INFO [RS:1;asf911:57040] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:30,466 INFO [RS:0;asf911:34337] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:30,467 INFO [M:1;asf911:46318] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:25:30,468 DEBUG [RS:1;asf911:57040] zookeeper.ZKUtil(355): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,468 DEBUG [RS:0;asf911:34337] zookeeper.ZKUtil(355): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,469 DEBUG [RS:1;asf911:57040] zookeeper.ZKUtil(355): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,469 DEBUG [RS:0;asf911:34337] zookeeper.ZKUtil(355): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,470 INFO [RS:1;asf911:57040] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2e503175 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:30,471 INFO [RS:0;asf911:34337] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x377d8602 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:30,491 DEBUG [RS:0;asf911:34337] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70950de5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:30,491 DEBUG [RS:0;asf911:34337] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:30,492 DEBUG [RS:1;asf911:57040] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d2722bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:30,492 INFO [RS:0;asf911:34337] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:30,492 DEBUG [RS:1;asf911:57040] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:30,495 INFO [RS:1;asf911:57040] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:30,512 INFO [RS:1;asf911:57040] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:30,512 INFO [RS:0;asf911:34337] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:30,513 INFO [RS:1;asf911:57040] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:30,513 INFO [RS:0;asf911:34337] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:30,513 INFO [RS:1;asf911:57040] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:30,514 INFO [RS:0;asf911:34337] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:30,517 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:57040, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,518 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:57040, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:30,518 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:34337, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,518 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:57040, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,518 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:34337, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:30,518 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:57040, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,518 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:34337, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,518 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:57040, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:30,519 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:57040, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:30,519 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:34337, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,519 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:57040, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:30,519 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:34337, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:30,520 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:57040, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,520 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:34337, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:30,520 DEBUG [RS:1;asf911:57040] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:57040, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:30,520 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:34337, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:30,520 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:34337, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:30,520 DEBUG [RS:0;asf911:34337] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:34337, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:30,546 INFO [RS:0;asf911:34337] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:30,547 INFO [SplitLogWorker-asf911:34337] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,34337,1515828329843 starting 2018-01-13 07:25:30,547 INFO [RS:1;asf911:57040] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:30,547 INFO [SplitLogWorker-asf911:57040] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,57040,1515828329910 starting 2018-01-13 07:25:30,573 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,34337,1515828329843, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:34337, sessionid=0x160ee6894500011 2018-01-13 07:25:30,573 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:30,573 DEBUG [RS:0;asf911:34337] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,573 INFO [RS:1;asf911:57040] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,57040,1515828329910, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:57040, sessionid=0x160ee6894500012 2018-01-13 07:25:30,573 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,34337,1515828329843' 2018-01-13 07:25:30,573 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:30,573 DEBUG [RS:1;asf911:57040] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,573 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,57040,1515828329910' 2018-01-13 07:25:30,574 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:30,573 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:30,574 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:30,574 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:30,575 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:30,575 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:30,575 DEBUG [RS:0;asf911:34337] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:30,575 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,34337,1515828329843' 2018-01-13 07:25:30,575 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:30,575 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:30,577 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:30,577 DEBUG [RS:1;asf911:57040] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:30,577 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,57040,1515828329910' 2018-01-13 07:25:30,578 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:30,578 DEBUG [RS:0;asf911:34337] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:30,578 DEBUG [RS:1;asf911:57040] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:30,579 DEBUG [RS:0;asf911:34337] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:30,579 DEBUG [RS:1;asf911:57040] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:30,579 INFO [RS:0;asf911:34337] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:30,579 INFO [RS:0;asf911:34337] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:30,579 INFO [RS:1;asf911:57040] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:30,579 INFO [RS:1;asf911:57040] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:30,660 DEBUG [M:1;asf911:46318] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:30,661 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:30,661 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:30,662 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:30,665 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:25:30,704 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:25:30,723 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:30,724 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:30,724 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:25:30,725 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:25:30,777 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:25:30,777 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:25:30,928 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:25:30,928 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:25:30,940 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:31,592 INFO [RS:0;asf911:34337] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C34337%2C1515828329843, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:31,593 INFO [RS:1;asf911:57040] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C57040%2C1515828329910, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,57040,1515828329910, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:31,626 DEBUG [RS-EventLoopGroup-11-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:31,627 DEBUG [RS-EventLoopGroup-11-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:31,627 DEBUG [RS-EventLoopGroup-11-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:31,627 DEBUG [RS-EventLoopGroup-11-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:31,639 INFO [RS:0;asf911:34337] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843/asf911.gq1.ygridcore.net%2C34337%2C1515828329843.1515828331598 2018-01-13 07:25:31,639 INFO [RS:1;asf911:57040] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,57040,1515828329910/asf911.gq1.ygridcore.net%2C57040%2C1515828329910.1515828331598 2018-01-13 07:25:31,639 DEBUG [RS:0;asf911:34337] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:25:31,640 DEBUG [RS:1;asf911:57040] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:25:31,642 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:31,648 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:25:31,657 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:31,808 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,34337,1515828329843 version=3145728 2018-01-13 07:25:31,809 DEBUG [RSProcedureDispatcher-pool13-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:31,810 DEBUG [RSProcedureDispatcher-pool13-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:31,817 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:31,817 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 46536 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:31,819 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34337] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:25:31,819 INFO [RS_OPEN_META-asf911:34337-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:31,822 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741825_1001, blk_1073741826_1002, blk_1073741827_1003, blk_1073741828_1004, blk_1073741829_1005, blk_1073741830_1006, blk_1073741831_1007, blk_1073741832_1008, blk_1073741833_1009, blk_1073741834_1010] 2018-01-13 07:25:31,827 INFO [RS_OPEN_META-asf911:34337-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C34337%2C1515828329843.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:31,926 DEBUG [RS-EventLoopGroup-11-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:31,927 DEBUG [RS-EventLoopGroup-11-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:32,036 INFO [RS_OPEN_META-asf911:34337-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843/asf911.gq1.ygridcore.net%2C34337%2C1515828329843.meta.1515828331828.meta 2018-01-13 07:25:32,036 DEBUG [RS_OPEN_META-asf911:34337-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:25:32,037 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:32,037 DEBUG [RS_OPEN_META-asf911:34337-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:25:32,038 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:25:32,039 INFO [RS_OPEN_META-asf911:34337-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:25:32,039 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:25:32,039 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:32,048 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:32,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:32,049 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:32,049 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:32,050 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:32,054 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:32,054 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:32,054 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:32,055 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:32,055 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:32,058 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:32,058 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:32,059 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:32,059 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:32,060 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:32,063 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:32,063 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:32,063 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:32,063 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:32,064 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:32,067 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:32,067 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:32,067 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:32,068 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:32,069 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:32,073 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:32,079 DEBUG [RS_OPEN_META-asf911:34337-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:32,084 DEBUG [RS_OPEN_META-asf911:34337-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:25:32,086 INFO [RS_OPEN_META-asf911:34337-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:25:32,090 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:25:32,096 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46318] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:32,096 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:25:32,097 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828332096}]},"ts":1515828332096} 2018-01-13 07:25:32,098 DEBUG [RS_OPEN_META-asf911:34337-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:32,103 DEBUG [hconnection-0x32cf1201-shared-pool16-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:32,110 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:32,111 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 46540 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:32,119 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:25:32,120 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:32,131 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:32,186 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:25:32,228 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:25:32,228 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.4070sec 2018-01-13 07:25:32,270 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.7610sec 2018-01-13 07:25:32,271 INFO [M:1;asf911:46318] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:32,282 INFO [M:1;asf911:46318] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34337,1515828329843 belongs to an existing region server 2018-01-13 07:25:32,283 INFO [M:1;asf911:46318] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,57040,1515828329910 belongs to an existing region server 2018-01-13 07:25:32,283 INFO [M:1;asf911:46318] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:25:32,287 INFO [M:1;asf911:46318] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:25:32,287 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828337287 2018-01-13 07:25:32,287 INFO [M:1;asf911:46318] assignment.AssignmentManager(1219): Joined the cluster in 4msec, failover=true 2018-01-13 07:25:32,292 INFO [M:1;asf911:46318] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:25:32,292 INFO [M:1;asf911:46318] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:25:32,436 DEBUG [M:1;asf911:46318] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:32,438 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:32,493 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:32,519 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:32,522 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:32,526 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:32,528 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:25:32,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:32,550 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741844_1020 size 42 2018-01-13 07:25:32,550 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:32,551 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.: disabling compactions & flushes 2018-01-13 07:25:32,551 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:32,551 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:32,634 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:32,636 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:25:32,651 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:25:32,714 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:32,715 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828332714}]},"ts":1515828332714} 2018-01-13 07:25:32,720 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:25:32,720 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:32,730 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843}] 2018-01-13 07:25:32,789 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843 hbase:namespace hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:32,792 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34337,1515828329843; forceNewPlan=false, retain=false 2018-01-13 07:25:32,945 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:25:32,948 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:32,949 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x3e15ace1 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:32,967 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30d25553, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:32,970 DEBUG [MultiHConnection-shared--pool17-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:32,974 INFO [RS-EventLoopGroup-10-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:32,974 INFO [RS-EventLoopGroup-10-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 46547 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:32,980 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843; rit=OPENING, location=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:33,131 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,34337,1515828329843 version=3145728 2018-01-13 07:25:33,133 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=34337] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:33,141 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] regionserver.HRegion(6949): Opening region: {ENCODED => 5e5df243b1d6b14f3ccd851af4dbf5cf, NAME => 'hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:33,141 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 5e5df243b1d6b14f3ccd851af4dbf5cf 2018-01-13 07:25:33,141 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:33,149 DEBUG [StoreOpener-5e5df243b1d6b14f3ccd851af4dbf5cf-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/5e5df243b1d6b14f3ccd851af4dbf5cf/info 2018-01-13 07:25:33,150 DEBUG [StoreOpener-5e5df243b1d6b14f3ccd851af4dbf5cf-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/5e5df243b1d6b14f3ccd851af4dbf5cf/info 2018-01-13 07:25:33,150 INFO [StoreOpener-5e5df243b1d6b14f3ccd851af4dbf5cf-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:33,150 INFO [StoreOpener-5e5df243b1d6b14f3ccd851af4dbf5cf-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:33,151 INFO [StoreOpener-5e5df243b1d6b14f3ccd851af4dbf5cf-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:33,153 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/5e5df243b1d6b14f3ccd851af4dbf5cf 2018-01-13 07:25:33,165 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/5e5df243b1d6b14f3ccd851af4dbf5cf/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:33,165 INFO [RS_OPEN_PRIORITY_REGION-asf911:34337-0] regionserver.HRegion(960): Onlined 5e5df243b1d6b14f3ccd851af4dbf5cf; next sequenceid=2 2018-01-13 07:25:33,170 INFO [PostOpenDeployTasks:5e5df243b1d6b14f3ccd851af4dbf5cf] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:33,173 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46318] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843; rit=OPENING, location=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:33,174 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:33,175 DEBUG [PostOpenDeployTasks:5e5df243b1d6b14f3ccd851af4dbf5cf] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:33,176 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:34337-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. on asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:33,244 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:33,298 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:25:33,298 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:33,298 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=5e5df243b1d6b14f3ccd851af4dbf5cf, target=asf911.gq1.ygridcore.net,34337,1515828329843 in 454msec 2018-01-13 07:25:33,298 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828333298}]},"ts":1515828333298} 2018-01-13 07:25:33,304 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:25:33,354 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:25:33,375 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:33,381 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:33,428 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.0830sec 2018-01-13 07:25:33,469 DEBUG [M:1;asf911:46318] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,470 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,519 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,569 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,610 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,623 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:33,677 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:33,719 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 288msec 2018-01-13 07:25:33,885 DEBUG [M:1;asf911:46318] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:33,886 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:33,944 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:34,002 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:34,060 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:34,073 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:34,146 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:34,202 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 421msec 2018-01-13 07:25:34,232 DEBUG [M:1;asf911:46318] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:25:34,240 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:34,248 DEBUG [M:1;asf911:46318] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:25:34,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:34,257 INFO [M:1;asf911:46318] master.HMaster(928): Master has completed initialization 4.283sec 2018-01-13 07:25:34,258 INFO [M:1;asf911:46318] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:25:34,258 INFO [M:1;asf911:46318] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:25:34,260 INFO [M:1;asf911:46318] regionserver.HRegionServer(866): ClusterId : b696352d-1ffd-427e-9e42-770be24a4b2e 2018-01-13 07:25:34,354 DEBUG [hconnection-0x51a18b21-shared-pool12-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:34,362 DEBUG [Default-IPC-NioEventLoopGroup-7-2] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:41843 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:41843 2018-01-13 07:25:34,507 DEBUG [hconnection-0x51a18b21-shared-pool12-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:34,512 INFO [RS-EventLoopGroup-10-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:34,512 INFO [RS-EventLoopGroup-10-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 46552 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:34,518 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:34,523 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testCreateSilentIsReallySilent connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:34,540 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testCreateSilentIsReallySilent0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:34,541 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testCreateSilentIsReallySilent-0x160ee689450001b connected 2018-01-13 07:25:34,591 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,34337,1515828329843: killAll ***** 2018-01-13 07:25:34,591 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:34,606 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1209532416, "init" : 792723456, "max" : 2609905664, "used" : 704133496 }, "NonHeapMemoryUsage" : { "committed" : 112263168, "init" : 2555904, "max" : -1, "used" : 110594776 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 5541, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 29, "TotalCallTime_min" : 0, "TotalCallTime_max" : 58, "TotalCallTime_mean" : 5, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 6, "TotalCallTime_90th_percentile" : 12, "TotalCallTime_95th_percentile" : 31, "TotalCallTime_98th_percentile" : 58, "TotalCallTime_99th_percentile" : 58, "TotalCallTime_99.9th_percentile" : 58, "TotalCallTime_TimeRangeCount_0-1" : 14, "TotalCallTime_TimeRangeCount_1-3" : 5, "TotalCallTime_TimeRangeCount_3-10" : 7, "TotalCallTime_TimeRangeCount_10-30" : 1, "TotalCallTime_TimeRangeCount_30-100" : 2, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 29, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 57, "ResponseSize_90th_percentile" : 117, "ResponseSize_95th_percentile" : 425, "ResponseSize_98th_percentile" : 694, "ResponseSize_99th_percentile" : 783, "ResponseSize_99.9th_percentile" : 864, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 29, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 58, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 5, "ProcessCallTime_90th_percentile" : 12, "ProcessCallTime_95th_percentile" : 31, "ProcessCallTime_98th_percentile" : 58, "ProcessCallTime_99th_percentile" : 58, "ProcessCallTime_99.9th_percentile" : 58, "ProcessCallTime_TimeRangeCount_0-1" : 14, "ProcessCallTime_TimeRangeCount_1-3" : 5, "ProcessCallTime_TimeRangeCount_3-10" : 7, "ProcessCallTime_TimeRangeCount_10-30" : 1, "ProcessCallTime_TimeRangeCount_30-100" : 2, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 4, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 29, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 87, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 129, "RequestSize_90th_percentile" : 272, "RequestSize_95th_percentile" : 504, "RequestSize_98th_percentile" : 644, "RequestSize_99th_percentile" : 690, "RequestSize_99.9th_percentile" : 732, "RequestSize_SizeRangeCount_10-100" : 15, "RequestSize_SizeRangeCount_100-1000" : 14, "sentBytes" : 2642, "QueueCallTime_num_ops" : 29, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 0, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:34,611 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46318] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,34337,1515828329843 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,34337,1515828329843: killAll ***** 2018-01-13 07:25:34,612 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,34337,1515828329843' ***** 2018-01-13 07:25:34,613 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:34,613 INFO [RS:0;asf911:34337] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:34,613 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,57040,1515828329910: killAll ***** 2018-01-13 07:25:34,614 INFO [SplitLogWorker-asf911:34337] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:34,614 INFO [RS:0;asf911:34337] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:34,614 INFO [SplitLogWorker-asf911:34337] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,34337,1515828329843 exiting 2018-01-13 07:25:34,614 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:34,615 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:34,614 INFO [RS:0;asf911:34337] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:34,614 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:34,616 INFO [RS:0;asf911:34337] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:34,617 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:34,617 DEBUG [RS:0;asf911:34337] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:34,617 DEBUG [RS_CLOSE_REGION-asf911:34337-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:34,617 INFO [RS:0;asf911:34337] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5f401d09 to localhost:59094 2018-01-13 07:25:34,617 DEBUG [RS:0;asf911:34337] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,618 DEBUG [RS_CLOSE_REGION-asf911:34337-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.: disabling compactions & flushes 2018-01-13 07:25:34,618 INFO [RS:0;asf911:34337] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:34,618 INFO [RS:0;asf911:34337] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:34,618 INFO [RS:0;asf911:34337] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:34,618 DEBUG [RS_CLOSE_REGION-asf911:34337-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:34,619 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:25:34,619 DEBUG [RS_CLOSE_META-asf911:34337-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:25:34,619 DEBUG [RS:0;asf911:34337] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740, 5e5df243b1d6b14f3ccd851af4dbf5cf=hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.} 2018-01-13 07:25:34,620 DEBUG [RS_CLOSE_META-asf911:34337-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:34,620 DEBUG [RS_CLOSE_META-asf911:34337-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:34,620 INFO [StoreCloserThread-hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf.-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:34,620 ERROR [RS_CLOSE_REGION-asf911:34337-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:25:34,621 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:34,621 INFO [RS_CLOSE_REGION-asf911:34337-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:34,621 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:34,621 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1209532416, "init" : 792723456, "max" : 2609905664, "used" : 713560688 }, "NonHeapMemoryUsage" : { "committed" : 112394240, "init" : 2555904, "max" : -1, "used" : 110816616 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 5541, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 29, "TotalCallTime_min" : 0, "TotalCallTime_max" : 58, "TotalCallTime_mean" : 5, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 6, "TotalCallTime_90th_percentile" : 12, "TotalCallTime_95th_percentile" : 31, "TotalCallTime_98th_percentile" : 58, "TotalCallTime_99th_percentile" : 58, "TotalCallTime_99.9th_percentile" : 58, "TotalCallTime_TimeRangeCount_0-1" : 14, "TotalCallTime_TimeRangeCount_1-3" : 5, "TotalCallTime_TimeRangeCount_3-10" : 7, "TotalCallTime_TimeRangeCount_10-30" : 1, "TotalCallTime_TimeRangeCount_30-100" : 2, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 29, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 57, "ResponseSize_90th_percentile" : 117, "ResponseSize_95th_percentile" : 425, "ResponseSize_98th_percentile" : 694, "ResponseSize_99th_percentile" : 783, "ResponseSize_99.9th_percentile" : 864, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 29, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 58, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 5, "ProcessCallTime_90th_percentile" : 12, "ProcessCallTime_95th_percentile" : 31, "ProcessCallTime_98th_percentile" : 58, "ProcessCallTime_99th_percentile" : 58, "ProcessCallTime_99.9th_percentile" : 58, "ProcessCallTime_TimeRangeCount_0-1" : 14, "ProcessCallTime_TimeRangeCount_1-3" : 5, "ProcessCallTime_TimeRangeCount_3-10" : 7, "ProcessCallTime_TimeRangeCount_10-30" : 1, "ProcessCallTime_TimeRangeCount_30-100" : 2, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 4, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 29, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 87, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 129, "RequestSize_90th_percentile" : 272, "RequestSize_95th_percentile" : 504, "RequestSize_98th_percentile" : 644, "RequestSize_99th_percentile" : 690, "RequestSize_99.9th_percentile" : 732, "RequestSize_SizeRangeCount_10-100" : 15, "RequestSize_SizeRangeCount_100-1000" : 14, "sentBytes" : 2642, "QueueCallTime_num_ops" : 29, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 0, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:34,621 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:34,621 DEBUG [RS_CLOSE_REGION-asf911:34337-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828332292.5e5df243b1d6b14f3ccd851af4dbf5cf. 2018-01-13 07:25:34,623 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:34,625 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:34,625 ERROR [RS_CLOSE_META-asf911:34337-0] regionserver.HRegion(1620): Memstore size is 1330 2018-01-13 07:25:34,627 DEBUG [RS_CLOSE_META-asf911:34337-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:34,627 INFO [RS_CLOSE_META-asf911:34337-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:34,627 ERROR [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46318] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,57040,1515828329910 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,57040,1515828329910: killAll ***** 2018-01-13 07:25:34,627 DEBUG [RS_CLOSE_META-asf911:34337-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:34,628 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,57040,1515828329910' ***** 2018-01-13 07:25:34,628 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:34,628 INFO [RS:1;asf911:57040] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:34,628 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:34,629 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,59777,1515828329677' ***** 2018-01-13 07:25:34,629 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:34,628 INFO [SplitLogWorker-asf911:57040] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:34,628 INFO [RS:1;asf911:57040] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:34,629 INFO [SplitLogWorker-asf911:57040] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,57040,1515828329910 exiting 2018-01-13 07:25:34,630 INFO [RS:1;asf911:57040] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:34,631 INFO [RS:1;asf911:57040] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:34,631 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:34,631 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:34,630 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:34,632 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:34,632 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,46318,1515828329779' ***** 2018-01-13 07:25:34,633 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:34,631 INFO [RS:1;asf911:57040] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:34,634 DEBUG [RS:1;asf911:57040] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:34,631 INFO [M:0;asf911:59777] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x326c560d to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:34,634 DEBUG [M:1;asf911:46318] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b05db5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:34,634 INFO [RS:1;asf911:57040] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x01853a41 to localhost:59094 2018-01-13 07:25:34,634 DEBUG [RS:1;asf911:57040] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,634 INFO [M:1;asf911:46318] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:34,635 DEBUG [M:1;asf911:46318] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:34,635 INFO [RS:1;asf911:57040] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,57040,1515828329910; all regions closed. 2018-01-13 07:25:34,639 INFO [M:1;asf911:46318] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x78432573 to localhost:59094 2018-01-13 07:25:34,643 DEBUG [M:1;asf911:46318] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,643 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:34,643 INFO [M:1;asf911:46318] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,46318,1515828329779; all regions closed. 2018-01-13 07:25:34,643 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:25:34,643 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:34,643 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:34,643 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:34,644 DEBUG [M:1;asf911:46318] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,646 INFO [M:1;asf911:46318] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,46318,1515828329779 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46318,1515828329779-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46318,1515828329779-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46318,1515828329779-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:46318 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46318,1515828329779-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46318,1515828329779-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:34,647 INFO [M:1;asf911:46318] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:25:34,647 INFO [M:1;asf911:46318] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:25:34,647 DEBUG [M:1;asf911:46318] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:34,650 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:34,652 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 96 2018-01-13 07:25:34,652 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,57040,1515828329910/asf911.gq1.ygridcore.net%2C57040%2C1515828329910.1515828331598 not finished, retry = 0 2018-01-13 07:25:34,653 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741840_1016 size 96 2018-01-13 07:25:34,656 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:34,656 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:34,660 DEBUG [M:1;asf911:46318] zookeeper.ZKUtil(614): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:34,660 DEBUG [main] zookeeper.ZKUtil(614): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:34,660 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:34,660 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:34,661 DEBUG [M:0;asf911:59777] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cabae30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:34,660 WARN [M:1;asf911:46318] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:34,661 INFO [M:1;asf911:46318] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:25:34,661 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,46318,1515828329779 2018-01-13 07:25:34,661 INFO [M:1;asf911:46318] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x3e15ace1 to localhost:59094 2018-01-13 07:25:34,661 DEBUG [M:1;asf911:46318] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,662 INFO [M:1;asf911:46318] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:25:34,662 INFO [M:1;asf911:46318] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:25:34,663 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,663 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,665 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,663 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,664 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,670 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,664 INFO [M:1;asf911:46318] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:25:34,670 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,670 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,670 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:34,667 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:34,667 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:34,667 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:34,673 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:34,675 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:46318-0x160ee6894500010, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:34,673 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:34,675 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:34,678 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:25:34,678 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:25:34,680 INFO [M:1;asf911:46318] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,46318,1515828329779_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:34,680 INFO [M:1;asf911:46318] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:25:34,680 INFO [M:1;asf911:46318] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:34,690 DEBUG [M:1;asf911:46318] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,46318,1515828329779 already deleted, retry=false 2018-01-13 07:25:34,698 INFO [M:1;asf911:46318] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,46318,1515828329779; zookeeper connection closed. 2018-01-13 07:25:34,698 INFO [M:1;asf911:46318] regionserver.HRegionServer(1157): M:1;asf911:46318 exiting 2018-01-13 07:25:34,756 DEBUG [RS:1;asf911:57040] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,756 INFO [RS:1;asf911:57040] regionserver.Leases(147): RS:1;asf911:57040 closing leases 2018-01-13 07:25:34,756 INFO [RS:1;asf911:57040] regionserver.Leases(150): RS:1;asf911:57040 closed leases 2018-01-13 07:25:34,757 INFO [RS:1;asf911:57040] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,57040,1515828329910 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,57040,1515828329910 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:34,757 INFO [RS:1;asf911:57040] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:34,757 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:34,757 INFO [RS:1;asf911:57040] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:34,758 INFO [RS:1;asf911:57040] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:34,759 INFO [RS:1;asf911:57040] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:34,765 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:34,765 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,57040,1515828329910 2018-01-13 07:25:34,765 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:57040-0x160ee6894500012, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:34,765 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:34,773 INFO [RS:1;asf911:57040] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,57040,1515828329910; zookeeper connection closed. 2018-01-13 07:25:34,773 INFO [RS:1;asf911:57040] regionserver.HRegionServer(1157): RS:1;asf911:57040 exiting 2018-01-13 07:25:34,774 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5bc62e44] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5bc62e44 2018-01-13 07:25:34,820 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,34337,1515828329843; all regions closed. 2018-01-13 07:25:34,822 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741825_1001, blk_1073741826_1002, blk_1073741827_1003, blk_1073741828_1004, blk_1073741829_1005, blk_1073741830_1006, blk_1073741831_1007, blk_1073741832_1008, blk_1073741833_1009, blk_1073741834_1010] 2018-01-13 07:25:34,826 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:34,826 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:34,835 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:34,835 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:34,836 DEBUG [RS:0;asf911:34337] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:34,836 INFO [RS:0;asf911:34337] regionserver.Leases(147): RS:0;asf911:34337 closing leases 2018-01-13 07:25:34,836 INFO [RS:0;asf911:34337] regionserver.Leases(150): RS:0;asf911:34337 closed leases 2018-01-13 07:25:34,837 INFO [RS:0;asf911:34337] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,34337,1515828329843 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,34337,1515828329843 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:34,837 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:34,837 INFO [RS:0;asf911:34337] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:34,848 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34337-0x160ee6894500011, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34337,1515828329843 2018-01-13 07:25:34,857 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,34337,1515828329843; zookeeper connection closed. 2018-01-13 07:25:34,857 INFO [RS:0;asf911:34337] regionserver.HRegionServer(1157): RS:0;asf911:34337 exiting 2018-01-13 07:25:34,857 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1f19386] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1f19386 2018-01-13 07:25:34,857 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:25:35,522 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:35,522 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:35,524 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:35,524 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:35,663 INFO [M:0;asf911:59777] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:35,665 INFO [M:0;asf911:59777] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,59777,1515828329677 2018-01-13 07:25:35,665 DEBUG [M:0;asf911:59777] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:35,665 INFO [M:0;asf911:59777] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x326c560d to localhost:59094 2018-01-13 07:25:35,665 DEBUG [M:0;asf911:59777] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:35,665 INFO [M:0;asf911:59777] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,59777,1515828329677; all regions closed. 2018-01-13 07:25:35,665 INFO [M:0;asf911:59777] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,59777,1515828329677 had [] on shutdown 2018-01-13 07:25:35,665 DEBUG [M:0;asf911:59777] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:35,673 DEBUG [M:0;asf911:59777] zookeeper.ZKUtil(614): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:35,673 WARN [M:0;asf911:59777] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:35,682 DEBUG [M:0;asf911:59777] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,59777,1515828329677 already deleted, retry=false 2018-01-13 07:25:35,690 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:59777-0x160ee689450000f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,59777,1515828329677 2018-01-13 07:25:35,690 INFO [M:0;asf911:59777] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,59777,1515828329677; zookeeper connection closed. 2018-01-13 07:25:35,690 INFO [M:0;asf911:59777] regionserver.HRegionServer(1157): M:0;asf911:59777 exiting 2018-01-13 07:25:35,699 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:35,703 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741839_1015 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:35,704 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741841_1017 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,704 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741842_1018 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,704 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741840_1016 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,705 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:35,705 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,705 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741843_1019 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,706 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741844_1020 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,706 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,706 INFO [IPC Server handler 0 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741835_1011 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:35,707 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:35,715 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:35,716 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee689450001e connected 2018-01-13 07:25:35,717 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:25:35,718 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:25:35,719 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:25:35,720 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:25:35,720 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:25:35,721 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:25:35,722 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:25:35,723 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:25:35,724 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:25:35,724 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:25:35,725 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:25:35,726 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:25:35,727 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:25:35,731 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:25:35,731 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:25:35,732 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:25:35,734 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:25:35,734 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:25:35,735 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:25:35,737 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:25:35,738 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:25:35,773 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:35,773 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:25:35,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:35,775 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:25:35,776 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:35,779 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:25:35,779 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:25:35,780 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:35,780 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:35,780 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:35,780 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450001e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:25:35,812 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testCreateSilentIsReallySilent Thread=227 (was 174) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-11-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-154731486_1 at /127.0.0.1:54396 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:46318-HFileCleaner.small.0-1515828330313 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-11-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:46318-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-11-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:46318-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:46318-HFileCleaner.large.0-1515828330313 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-11-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=1655 (was 1239) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=208 (was 208), ProcessCount=271 (was 270) - ProcessCount LEAK? -, AvailableMemoryMB=10982 (was 11149) 2018-01-13 07:25:35,813 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1655 is superior to 1024 2018-01-13 07:25:35,835 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testZNodeDeletes Thread=225, OpenFileDescriptor=1651, MaxFileDescriptor=60000, SystemLoadAverage=208, ProcessCount=271, AvailableMemoryMB=10979 2018-01-13 07:25:35,836 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1651 is superior to 1024 2018-01-13 07:25:35,855 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:35,856 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:35,859 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:25:35,859 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:25:35,863 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:35,863 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:35,863 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:35,863 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:35,864 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:35,865 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:33746 2018-01-13 07:25:35,865 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:35,866 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:35,867 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:35,869 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:35,870 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:33746 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:35,882 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:337460x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:35,883 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:33746-0x160ee689450001f connected 2018-01-13 07:25:35,949 DEBUG [main] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:35,950 DEBUG [main] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:35,950 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=33746 2018-01-13 07:25:35,951 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=33746 2018-01-13 07:25:35,953 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=33746 2018-01-13 07:25:35,954 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:35,956 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:35,957 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:35,957 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:35,957 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:35,957 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:35,959 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:42367 2018-01-13 07:25:35,959 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:35,960 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:35,960 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:35,962 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:35,963 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:42367 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:35,974 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:423670x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:35,974 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:42367-0x160ee6894500020 connected 2018-01-13 07:25:35,982 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:25:35,987 DEBUG [main] zookeeper.ZKUtil(357): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:35,988 DEBUG [main] zookeeper.ZKUtil(357): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:35,988 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=42367 2018-01-13 07:25:35,989 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=42367 2018-01-13 07:25:35,990 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=42367 2018-01-13 07:25:35,991 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:36,016 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:36,016 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:36,016 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:36,017 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:36,017 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:36,017 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:36,018 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:53530 2018-01-13 07:25:36,018 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,019 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,020 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:36,021 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:36,023 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:53530 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:36,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:535300x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:36,083 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:53530-0x160ee6894500021 connected 2018-01-13 07:25:36,083 DEBUG [main] zookeeper.ZKUtil(357): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:36,084 DEBUG [main] zookeeper.ZKUtil(357): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:36,084 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=53530 2018-01-13 07:25:36,086 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=53530 2018-01-13 07:25:36,086 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=53530 2018-01-13 07:25:36,112 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:36,112 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:36,112 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:36,112 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:36,112 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:36,113 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:36,114 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:43833 2018-01-13 07:25:36,114 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,114 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,115 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:36,117 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:36,118 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:43833 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:36,132 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:438330x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:36,133 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:43833-0x160ee6894500022 connected 2018-01-13 07:25:36,133 DEBUG [main] zookeeper.ZKUtil(357): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:36,134 DEBUG [main] zookeeper.ZKUtil(357): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:36,134 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=43833 2018-01-13 07:25:36,135 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=43833 2018-01-13 07:25:36,136 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=43833 2018-01-13 07:25:36,138 INFO [M:0;asf911:33746] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,139 INFO [M:1;asf911:42367] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,42367,1515828335955 2018-01-13 07:25:36,149 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(355): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,149 DEBUG [M:1;asf911:42367] zookeeper.ZKUtil(355): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,42367,1515828335955 2018-01-13 07:25:36,156 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:36,157 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:36,158 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:36,158 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:36,165 DEBUG [M:1;asf911:42367] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:25:36,165 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(355): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:36,169 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:36,169 DEBUG [M:1;asf911:42367] zookeeper.ZKUtil(355): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:36,169 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:36,169 INFO [M:0;asf911:33746] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,33746,1515828335861 from backup master directory 2018-01-13 07:25:36,169 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:36,170 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:36,170 INFO [M:1;asf911:42367] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,33746,1515828335861; waiting to become the next active master 2018-01-13 07:25:36,181 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,181 WARN [M:0;asf911:33746] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:36,181 INFO [M:0;asf911:33746] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,207 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:36,207 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:36,210 DEBUG [M:0;asf911:33746] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 29c9e3e4-d0a2-4f3d-b043-e485da8cb2ce 2018-01-13 07:25:36,216 INFO [M:0;asf911:33746] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:25:36,217 INFO [M:0;asf911:33746] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:25:36,234 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:36,234 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:36,236 DEBUG [M:0;asf911:33746] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:36,243 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:36,243 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:36,243 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,244 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:36,249 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:36,249 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:36,249 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,249 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:36,253 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:36,254 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:36,254 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,254 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:36,258 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:36,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:36,259 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,259 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:36,263 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:36,264 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:36,264 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:36,264 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:36,266 DEBUG [M:0;asf911:33746] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:36,270 DEBUG [M:0;asf911:33746] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:36,275 DEBUG [M:0;asf911:33746] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:36,275 INFO [M:0;asf911:33746] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:25:36,275 DEBUG [M:0;asf911:33746] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:36,275 DEBUG [M:0;asf911:33746] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:36,276 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:36,277 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:36,277 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:36,277 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:36,277 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:36,277 INFO [M:0;asf911:33746] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:36,292 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:36,293 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741848_1024 size 2318 2018-01-13 07:25:36,295 DEBUG [M:0;asf911:33746] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:36,316 INFO [M:0;asf911:33746] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:36,317 INFO [M:0;asf911:33746] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:25:36,317 DEBUG [M:0;asf911:33746] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:25:36,334 INFO [M:0;asf911:33746] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x100d5392 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:36,343 DEBUG [M:0;asf911:33746] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@767975ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:36,364 INFO [M:0;asf911:33746] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:25:36,365 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:25:36,366 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:25:36,374 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:25:36,375 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:25:36,375 INFO [M:0;asf911:33746] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:25:36,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:36,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:36,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:36,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:36,382 INFO [M:0;asf911:33746] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,33746,1515828335861, sessionid=0x160ee689450001f, setting cluster-up flag (Was=false) 2018-01-13 07:25:36,448 INFO [M:0;asf911:33746] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:25:36,450 DEBUG [M:0;asf911:33746] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,490 INFO [M:0;asf911:33746] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:25:36,492 DEBUG [M:0;asf911:33746] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:36,493 INFO [M:0;asf911:33746] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:25:36,493 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:33746, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:36,494 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:33746, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:36,494 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:33746, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:36,494 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:33746, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:36,494 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:33746, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:36,494 DEBUG [M:0;asf911:33746] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:33746, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:36,497 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:25:36,497 INFO [M:0;asf911:33746] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:25:36,501 DEBUG [M:0;asf911:33746] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:25:36,502 INFO [M:0;asf911:33746] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:25:36,502 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 5msec 2018-01-13 07:25:36,502 DEBUG [M:0;asf911:33746] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:25:36,502 DEBUG [M:0;asf911:33746] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:25:36,502 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:25:36,505 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828366504 2018-01-13 07:25:36,505 INFO [M:0;asf911:33746] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:25:36,505 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:25:36,506 INFO [M:0;asf911:33746] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:36,506 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:25:36,506 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:25:36,507 INFO [M:0;asf911:33746] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:25:36,509 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:25:36,509 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:25:36,509 DEBUG [M:0;asf911:33746] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:25:36,510 DEBUG [M:0;asf911:33746] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:33746-HFileCleaner.large.0-1515828336510,5,main] 2018-01-13 07:25:36,510 DEBUG [M:0;asf911:33746] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:33746-HFileCleaner.small.0-1515828336510,5,main] 2018-01-13 07:25:36,510 INFO [M:0;asf911:33746] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828336149, completionTime=-1 2018-01-13 07:25:36,510 INFO [M:0;asf911:33746] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:25:36,515 DEBUG [M:0;asf911:33746-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:36,516 DEBUG [M:0;asf911:33746-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500024 connected 2018-01-13 07:25:36,544 INFO [RS:0;asf911:53530] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x305d907f to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:36,544 INFO [RS:1;asf911:43833] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x4bffc363 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:36,566 DEBUG [RS:0;asf911:53530] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e9a7804, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:36,567 DEBUG [RS:1;asf911:43833] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4538246c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:36,568 INFO [RS:0;asf911:53530] regionserver.HRegionServer(866): ClusterId : 29c9e3e4-d0a2-4f3d-b043-e485da8cb2ce 2018-01-13 07:25:36,568 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:36,619 DEBUG [RS:0;asf911:53530] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:36,619 INFO [RS:1;asf911:43833] regionserver.HRegionServer(866): ClusterId : 29c9e3e4-d0a2-4f3d-b043-e485da8cb2ce 2018-01-13 07:25:36,619 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:36,632 DEBUG [RS:1;asf911:43833] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:36,632 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:36,632 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:36,640 DEBUG [RS:0;asf911:53530] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:36,640 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:36,640 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:36,648 DEBUG [RS:1;asf911:43833] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:36,648 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:36,649 DEBUG [RS:0;asf911:53530] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3aa5fb0f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:36,649 DEBUG [RS:0;asf911:53530] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:53530 2018-01-13 07:25:36,649 INFO [RS:0;asf911:53530] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:36,649 INFO [RS:0;asf911:53530] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:36,650 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:36,650 DEBUG [RS:1;asf911:43833] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4107a813, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:36,650 INFO [RS:0;asf911:53530] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,33746,1515828335861 with port=53530, startcode=1515828336015 2018-01-13 07:25:36,650 DEBUG [RS:1;asf911:43833] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:43833 2018-01-13 07:25:36,650 INFO [RS:1;asf911:43833] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:36,651 INFO [RS:1;asf911:43833] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:36,651 DEBUG [RS:0;asf911:53530] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:33746 2018-01-13 07:25:36,651 INFO [RS:1;asf911:43833] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,33746,1515828335861 with port=43833, startcode=1515828336110 2018-01-13 07:25:36,653 DEBUG [RS:1;asf911:43833] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:33746 2018-01-13 07:25:36,657 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.4 (auth:SIMPLE) 2018-01-13 07:25:36,658 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.5 (auth:SIMPLE) 2018-01-13 07:25:36,659 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 44709 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:36,659 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40676 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:36,662 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=33746] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,662 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=33746] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,664 DEBUG [RS:1;asf911:43833] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:36,664 DEBUG [RS:0;asf911:53530] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:36,665 DEBUG [RS:1;asf911:43833] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:36,665 DEBUG [RS:0;asf911:53530] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:36,666 DEBUG [RS:1;asf911:43833] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:36,666 DEBUG [RS:0;asf911:53530] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:36,673 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:36,682 DEBUG [RS:1;asf911:43833] zookeeper.ZKUtil(355): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,682 WARN [RS:1;asf911:43833] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:36,682 DEBUG [RS:0;asf911:53530] zookeeper.ZKUtil(355): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,682 DEBUG [RS:1;asf911:43833] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,682 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,682 WARN [RS:0;asf911:53530] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:36,683 DEBUG [RS:0;asf911:53530] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,691 INFO [RS:0;asf911:53530] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:36,691 INFO [RS:1;asf911:43833] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:36,693 DEBUG [RS:1;asf911:43833] zookeeper.ZKUtil(355): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,694 DEBUG [RS:0;asf911:53530] zookeeper.ZKUtil(355): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,694 DEBUG [RS:1;asf911:43833] zookeeper.ZKUtil(355): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,694 DEBUG [RS:0;asf911:53530] zookeeper.ZKUtil(355): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,696 INFO [RS:1;asf911:43833] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x497029f7 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:36,696 INFO [RS:0;asf911:53530] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5bfa2aba to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:36,712 INFO [M:0;asf911:33746] master.ServerManager(891): Finished wait on RegionServer count=2; waited=202ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:25:36,716 DEBUG [RS:1;asf911:43833] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@642ad118, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:36,716 DEBUG [RS:1;asf911:43833] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:36,716 INFO [M:0;asf911:33746] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:25:36,717 INFO [RS:1;asf911:43833] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:36,718 DEBUG [RS:0;asf911:53530] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f888d70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:36,718 DEBUG [RS:0;asf911:53530] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:36,718 INFO [RS:0;asf911:53530] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:36,723 INFO [RS:1;asf911:43833] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:36,723 INFO [RS:0;asf911:53530] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:36,723 INFO [RS:1;asf911:43833] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:36,724 INFO [RS:0;asf911:53530] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:36,724 INFO [RS:1;asf911:43833] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:36,725 INFO [RS:0;asf911:53530] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:36,729 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:43833, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:53530, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:43833, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:36,730 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:53530, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:36,730 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:43833, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:53530, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:43833, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:53530, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,730 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:43833, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:36,731 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:53530, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:36,731 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:43833, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:36,731 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:53530, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:36,731 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:43833, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:36,731 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:53530, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:36,731 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:43833, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,731 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:53530, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:36,732 DEBUG [RS:1;asf911:43833] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:43833, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:36,732 DEBUG [RS:0;asf911:53530] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:53530, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:36,762 INFO [RS:1;asf911:43833] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:36,762 INFO [RS:0;asf911:53530] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:36,762 INFO [SplitLogWorker-asf911:43833] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,43833,1515828336110 starting 2018-01-13 07:25:36,762 INFO [SplitLogWorker-asf911:53530] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,53530,1515828336015 starting 2018-01-13 07:25:36,783 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,53530,1515828336015, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:53530, sessionid=0x160ee6894500021 2018-01-13 07:25:36,784 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:36,784 DEBUG [RS:0;asf911:53530] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,784 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,53530,1515828336015' 2018-01-13 07:25:36,784 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,53530,1515828336015' 2018-01-13 07:25:36,785 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:36,786 DEBUG [RS:0;asf911:53530] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:36,786 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,43833,1515828336110, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:43833, sessionid=0x160ee6894500022 2018-01-13 07:25:36,786 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:36,786 DEBUG [RS:1;asf911:43833] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,786 DEBUG [RS:0;asf911:53530] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:36,787 INFO [RS:0;asf911:53530] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:36,787 INFO [RS:0;asf911:53530] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:36,786 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,43833,1515828336110' 2018-01-13 07:25:36,787 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:36,787 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:36,788 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:36,788 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:36,788 DEBUG [RS:1;asf911:43833] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:36,788 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,43833,1515828336110' 2018-01-13 07:25:36,788 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:36,789 DEBUG [RS:1;asf911:43833] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:36,789 DEBUG [RS:1;asf911:43833] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:36,789 INFO [RS:1;asf911:43833] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:36,789 INFO [RS:1;asf911:43833] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:36,896 DEBUG [M:0;asf911:33746] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:36,897 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:36,898 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:36,898 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:36,901 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:25:36,949 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:36,950 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:36,950 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:25:36,950 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:25:36,991 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:25:36,992 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:25:37,142 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:25:37,143 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:25:37,150 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:37,544 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:25:37,793 INFO [RS:0;asf911:53530] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C53530%2C1515828336015, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,53530,1515828336015, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:37,796 INFO [RS:1;asf911:43833] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C43833%2C1515828336110, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:37,811 DEBUG [RS-EventLoopGroup-15-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:25:37,811 DEBUG [RS-EventLoopGroup-15-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:37,815 DEBUG [RS-EventLoopGroup-15-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:37,815 DEBUG [RS-EventLoopGroup-15-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:37,819 INFO [RS:0;asf911:53530] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,53530,1515828336015/asf911.gq1.ygridcore.net%2C53530%2C1515828336015.1515828337795 2018-01-13 07:25:37,820 DEBUG [RS:0;asf911:53530] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:25:37,820 INFO [RS:1;asf911:43833] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110/asf911.gq1.ygridcore.net%2C43833%2C1515828336110.1515828337797 2018-01-13 07:25:37,820 DEBUG [RS:1;asf911:43833] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:25:37,823 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741840_1016, blk_1073741841_1017, blk_1073741842_1018, blk_1073741843_1019, blk_1073741844_1020, blk_1073741835_1011, blk_1073741836_1012, blk_1073741837_1013, blk_1073741838_1014, blk_1073741839_1015] 2018-01-13 07:25:37,823 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:37,831 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:25:37,840 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:37,991 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,43833,1515828336110 version=3145728 2018-01-13 07:25:37,992 DEBUG [RSProcedureDispatcher-pool21-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:37,993 DEBUG [RSProcedureDispatcher-pool21-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:37,996 INFO [RS-EventLoopGroup-15-13] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:37,997 INFO [RS-EventLoopGroup-15-13] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51338 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:37,999 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=43833] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:25:38,000 INFO [RS_OPEN_META-asf911:43833-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:38,004 INFO [RS_OPEN_META-asf911:43833-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C43833%2C1515828336110.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:38,016 DEBUG [RS-EventLoopGroup-15-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:38,016 DEBUG [RS-EventLoopGroup-15-15] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:38,020 INFO [RS_OPEN_META-asf911:43833-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110/asf911.gq1.ygridcore.net%2C43833%2C1515828336110.meta.1515828338005.meta 2018-01-13 07:25:38,021 DEBUG [RS_OPEN_META-asf911:43833-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:25:38,021 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:38,022 DEBUG [RS_OPEN_META-asf911:43833-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:25:38,022 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:25:38,022 INFO [RS_OPEN_META-asf911:43833-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:25:38,023 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:25:38,023 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:38,028 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:38,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:38,029 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:38,029 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:38,030 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:38,033 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:38,033 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:38,033 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:38,034 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:38,035 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:38,038 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:38,038 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:38,038 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:38,038 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:38,039 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:38,042 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:38,042 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:38,043 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:38,043 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:38,043 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:38,046 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:38,046 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:38,047 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:38,047 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:38,047 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:38,051 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:38,055 DEBUG [RS_OPEN_META-asf911:43833-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:38,061 DEBUG [RS_OPEN_META-asf911:43833-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:25:38,062 INFO [RS_OPEN_META-asf911:43833-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:25:38,065 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:25:38,071 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=33746] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:38,072 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:25:38,074 DEBUG [RS_OPEN_META-asf911:43833-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:38,075 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828338072}]},"ts":1515828338072} 2018-01-13 07:25:38,079 DEBUG [hconnection-0x7d60f6aa-shared-pool24-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:38,082 INFO [RS-EventLoopGroup-15-19] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:38,086 INFO [RS-EventLoopGroup-15-19] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51342 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:38,092 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:25:38,095 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:38,107 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:38,156 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:25:38,206 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:25:38,207 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.1570sec 2018-01-13 07:25:38,256 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.4890sec 2018-01-13 07:25:38,258 INFO [M:0;asf911:33746] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:38,270 INFO [M:0;asf911:33746] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,43833,1515828336110 belongs to an existing region server 2018-01-13 07:25:38,271 INFO [M:0;asf911:33746] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,53530,1515828336015 belongs to an existing region server 2018-01-13 07:25:38,272 INFO [M:0;asf911:33746] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:25:38,274 INFO [M:0;asf911:33746] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:25:38,275 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828343274 2018-01-13 07:25:38,275 INFO [M:0;asf911:33746] assignment.AssignmentManager(1219): Joined the cluster in 3msec, failover=true 2018-01-13 07:25:38,279 INFO [M:0;asf911:33746] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:25:38,280 INFO [M:0;asf911:33746] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:25:38,429 DEBUG [M:0;asf911:33746] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:38,430 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:38,479 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:38,502 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741853_1029{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 476 2018-01-13 07:25:38,503 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741853_1029 size 476 2018-01-13 07:25:38,912 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:38,915 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:25:38,934 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741854_1030{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 42 2018-01-13 07:25:38,934 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741854_1030 size 42 2018-01-13 07:25:39,336 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,338 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.: disabling compactions & flushes 2018-01-13 07:25:39,338 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,338 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,422 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:39,424 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:25:39,430 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:25:39,482 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:39,482 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828339482}]},"ts":1515828339482} 2018-01-13 07:25:39,491 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:25:39,491 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,499 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015}] 2018-01-13 07:25:39,549 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015 hbase:namespace hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,552 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015; rit=OFFLINE, location=asf911.gq1.ygridcore.net,53530,1515828336015; forceNewPlan=false, retain=false 2018-01-13 07:25:39,705 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:25:39,708 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:39,709 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x73de3603 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:39,729 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d19c608, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:39,732 DEBUG [MultiHConnection-shared--pool25-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:39,734 INFO [RS-EventLoopGroup-15-21] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:39,736 INFO [RS-EventLoopGroup-15-21] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51355 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:39,740 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015; rit=OPENING, location=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:39,891 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,53530,1515828336015 version=3145728 2018-01-13 07:25:39,892 DEBUG [RSProcedureDispatcher-pool21-t2] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:39,893 DEBUG [RSProcedureDispatcher-pool21-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:53530 2018-01-13 07:25:39,896 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:39,896 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49147 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:39,898 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=53530] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,905 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] regionserver.HRegion(6949): Opening region: {ENCODED => 704b0ace2f8515d8152f1fa5455ac155, NAME => 'hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:39,905 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 704b0ace2f8515d8152f1fa5455ac155 2018-01-13 07:25:39,905 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,912 DEBUG [StoreOpener-704b0ace2f8515d8152f1fa5455ac155-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/704b0ace2f8515d8152f1fa5455ac155/info 2018-01-13 07:25:39,912 DEBUG [StoreOpener-704b0ace2f8515d8152f1fa5455ac155-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/704b0ace2f8515d8152f1fa5455ac155/info 2018-01-13 07:25:39,912 INFO [StoreOpener-704b0ace2f8515d8152f1fa5455ac155-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:39,912 INFO [StoreOpener-704b0ace2f8515d8152f1fa5455ac155-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:39,913 INFO [StoreOpener-704b0ace2f8515d8152f1fa5455ac155-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:39,916 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/704b0ace2f8515d8152f1fa5455ac155 2018-01-13 07:25:39,928 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/704b0ace2f8515d8152f1fa5455ac155/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:39,928 INFO [RS_OPEN_PRIORITY_REGION-asf911:53530-0] regionserver.HRegion(960): Onlined 704b0ace2f8515d8152f1fa5455ac155; next sequenceid=2 2018-01-13 07:25:39,930 INFO [PostOpenDeployTasks:704b0ace2f8515d8152f1fa5455ac155] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,939 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=33746] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015; rit=OPENING, location=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:39,940 DEBUG [PostOpenDeployTasks:704b0ace2f8515d8152f1fa5455ac155] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:39,940 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:39,943 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:53530-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. on asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:40,004 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:40,054 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:25:40,054 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:40,054 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=704b0ace2f8515d8152f1fa5455ac155, target=asf911.gq1.ygridcore.net,53530,1515828336015 in 456msec 2018-01-13 07:25:40,055 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828340054}]},"ts":1515828340054} 2018-01-13 07:25:40,059 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:25:40,124 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:40,155 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:25:40,172 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:40,179 DEBUG [M:0;asf911:33746] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:53530 2018-01-13 07:25:40,182 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:40,182 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49150 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:40,183 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.8440sec 2018-01-13 07:25:40,335 DEBUG [M:0;asf911:33746] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,337 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,390 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:40,435 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-13 07:25:40,444 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,494 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:40,560 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:40,610 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 376msec 2018-01-13 07:25:40,769 DEBUG [M:0;asf911:33746] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:40,771 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:40,819 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:40,823 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741840_1016, blk_1073741841_1017, blk_1073741842_1018, blk_1073741843_1019, blk_1073741844_1020, blk_1073741835_1011, blk_1073741836_1012, blk_1073741837_1013, blk_1073741838_1014, blk_1073741839_1015] 2018-01-13 07:25:40,869 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:40,921 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:40,932 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:40,985 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:41,035 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 364msec 2018-01-13 07:25:41,057 DEBUG [M:0;asf911:33746] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:25:41,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:41,073 DEBUG [M:0;asf911:33746] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:25:41,081 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:41,082 INFO [M:0;asf911:33746] master.HMaster(928): Master has completed initialization 4.901sec 2018-01-13 07:25:41,083 INFO [M:0;asf911:33746] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:25:41,083 INFO [M:0;asf911:33746] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:25:41,085 INFO [M:0;asf911:33746] regionserver.HRegionServer(866): ClusterId : 29c9e3e4-d0a2-4f3d-b043-e485da8cb2ce 2018-01-13 07:25:41,147 DEBUG [hconnection-0x51a18b21-shared-pool12-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:41,149 DEBUG [Default-IPC-NioEventLoopGroup-7-4] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:34337 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:34337 2018-01-13 07:25:41,254 DEBUG [hconnection-0x51a18b21-shared-pool12-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:41,256 INFO [RS-EventLoopGroup-15-24] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:41,257 INFO [RS-EventLoopGroup-15-24] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51363 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:41,261 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:41,262 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=org.apache.hadoop.hbase.TestZooKeeper connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:41,274 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:41,278 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a connected 2018-01-13 07:25:41,342 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2/l3 2018-01-13 07:25:41,344 DEBUG [main] zookeeper.ZKUtil(355): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/l1/l2/l3/l4 2018-01-13 07:25:41,348 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2/l3/l4 2018-01-13 07:25:41,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1/l2/l3 2018-01-13 07:25:41,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2/l3 2018-01-13 07:25:41,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/l1/l2 2018-01-13 07:25:41,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/l1/l2 2018-01-13 07:25:41,349 DEBUG [main] zookeeper.ZKUtil(704): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /l1/l2/l3/l4 because node does not exist (not necessarily an error) 2018-01-13 07:25:41,350 DEBUG [main] zookeeper.ZKUtil(446): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Unable to list children of znode /l1/l2 because node does not exist (not an error) 2018-01-13 07:25:41,357 DEBUG [main] zookeeper.ZKUtil(704): org.apache.hadoop.hbase.TestZooKeeper-0x160ee689450002a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /l1/l2 because node does not exist (not necessarily an error) 2018-01-13 07:25:41,357 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,53530,1515828336015: killAll ***** 2018-01-13 07:25:41,358 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:41,364 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 322985896 }, "NonHeapMemoryUsage" : { "committed" : 116195328, "init" : 2555904, "max" : -1, "used" : 114513856 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 11082, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 58, "TotalCallTime_min" : 0, "TotalCallTime_max" : 10, "TotalCallTime_mean" : 2, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 5, "TotalCallTime_95th_percentile" : 7, "TotalCallTime_98th_percentile" : 10, "TotalCallTime_99th_percentile" : 10, "TotalCallTime_99.9th_percentile" : 10, "TotalCallTime_TimeRangeCount_0-1" : 17, "TotalCallTime_TimeRangeCount_1-3" : 8, "TotalCallTime_TimeRangeCount_3-10" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 29, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 58, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 11, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 114, "ResponseSize_95th_percentile" : 767, "ResponseSize_98th_percentile" : 825, "ResponseSize_99th_percentile" : 849, "ResponseSize_99.9th_percentile" : 870, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_100000000-inf" : 29, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 58, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 9, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 5, "ProcessCallTime_95th_percentile" : 7, "ProcessCallTime_98th_percentile" : 9, "ProcessCallTime_99th_percentile" : 9, "ProcessCallTime_99.9th_percentile" : 9, "ProcessCallTime_TimeRangeCount_0-1" : 17, "ProcessCallTime_TimeRangeCount_1-3" : 8, "ProcessCallTime_TimeRangeCount_3-10" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 29, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 8, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 58, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 129, "RequestSize_90th_percentile" : 378, "RequestSize_95th_percentile" : 518, "RequestSize_98th_percentile" : 713, "RequestSize_99th_percentile" : 725, "RequestSize_99.9th_percentile" : 735, "RequestSize_SizeRangeCount_10-100" : 15, "RequestSize_SizeRangeCount_100-1000" : 14, "RequestSize_SizeRangeCount_100000000-inf" : 29, "sentBytes" : 5284, "QueueCallTime_num_ops" : 58, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "QueueCallTime_TimeRangeCount_600000-inf" : 29, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:41,366 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=33746] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,53530,1515828336015 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,53530,1515828336015: killAll ***** 2018-01-13 07:25:41,367 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,53530,1515828336015' ***** 2018-01-13 07:25:41,367 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:41,367 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,43833,1515828336110: killAll ***** 2018-01-13 07:25:41,367 INFO [RS:0;asf911:53530] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:41,367 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:41,368 INFO [SplitLogWorker-asf911:53530] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:41,368 INFO [SplitLogWorker-asf911:53530] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,53530,1515828336015 exiting 2018-01-13 07:25:41,368 INFO [RS:0;asf911:53530] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:41,368 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:41,369 INFO [RS:0;asf911:53530] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:41,370 INFO [RS:0;asf911:53530] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:41,369 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:41,370 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:41,370 DEBUG [RS:0;asf911:53530] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:41,370 DEBUG [RS_CLOSE_REGION-asf911:53530-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:41,371 INFO [RS:0;asf911:53530] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x305d907f to localhost:59094 2018-01-13 07:25:41,372 DEBUG [RS_CLOSE_REGION-asf911:53530-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.: disabling compactions & flushes 2018-01-13 07:25:41,372 DEBUG [RS:0;asf911:53530] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,372 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1388): Waiting on 1 regions to close 2018-01-13 07:25:41,372 DEBUG [RS:0;asf911:53530] regionserver.HRegionServer(1392): Online Regions={704b0ace2f8515d8152f1fa5455ac155=hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.} 2018-01-13 07:25:41,372 DEBUG [RS_CLOSE_REGION-asf911:53530-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:41,373 INFO [StoreCloserThread-hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155.-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:41,373 ERROR [RS_CLOSE_REGION-asf911:53530-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:25:41,373 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 323018720 }, "NonHeapMemoryUsage" : { "committed" : 116260864, "init" : 2555904, "max" : -1, "used" : 114521464 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 11082, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 58, "TotalCallTime_min" : 0, "TotalCallTime_max" : 10, "TotalCallTime_mean" : 2, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 5, "TotalCallTime_95th_percentile" : 7, "TotalCallTime_98th_percentile" : 10, "TotalCallTime_99th_percentile" : 10, "TotalCallTime_99.9th_percentile" : 10, "TotalCallTime_TimeRangeCount_0-1" : 17, "TotalCallTime_TimeRangeCount_1-3" : 8, "TotalCallTime_TimeRangeCount_3-10" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 29, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 58, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 11, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 114, "ResponseSize_95th_percentile" : 767, "ResponseSize_98th_percentile" : 825, "ResponseSize_99th_percentile" : 849, "ResponseSize_99.9th_percentile" : 870, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_100000000-inf" : 29, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 58, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 9, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 5, "ProcessCallTime_95th_percentile" : 7, "ProcessCallTime_98th_percentile" : 9, "ProcessCallTime_99th_percentile" : 9, "ProcessCallTime_99.9th_percentile" : 9, "ProcessCallTime_TimeRangeCount_0-1" : 17, "ProcessCallTime_TimeRangeCount_1-3" : 8, "ProcessCallTime_TimeRangeCount_3-10" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 29, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 8, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 58, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 129, "RequestSize_90th_percentile" : 378, "RequestSize_95th_percentile" : 518, "RequestSize_98th_percentile" : 713, "RequestSize_99th_percentile" : 725, "RequestSize_99.9th_percentile" : 735, "RequestSize_SizeRangeCount_10-100" : 15, "RequestSize_SizeRangeCount_100-1000" : 14, "RequestSize_SizeRangeCount_100000000-inf" : 29, "sentBytes" : 5284, "QueueCallTime_num_ops" : 58, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "QueueCallTime_TimeRangeCount_600000-inf" : 29, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:41,373 INFO [RS_CLOSE_REGION-asf911:53530-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:41,376 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=33746] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,43833,1515828336110 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,43833,1515828336110: killAll ***** 2018-01-13 07:25:41,376 DEBUG [RS_CLOSE_REGION-asf911:53530-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828338279.704b0ace2f8515d8152f1fa5455ac155. 2018-01-13 07:25:41,376 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,43833,1515828336110' ***** 2018-01-13 07:25:41,377 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:25:41,377 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:41,377 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:41,377 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,33746,1515828335861' ***** 2018-01-13 07:25:41,378 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:41,377 INFO [RS:1;asf911:43833] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:41,378 INFO [RS:1;asf911:43833] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:41,378 INFO [SplitLogWorker-asf911:43833] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:41,378 INFO [RS:1;asf911:43833] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:41,378 INFO [RS:1;asf911:43833] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:41,380 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:41,380 DEBUG [RS:1;asf911:43833] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:41,378 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:41,378 INFO [SplitLogWorker-asf911:43833] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,43833,1515828336110 exiting 2018-01-13 07:25:41,378 DEBUG [M:0;asf911:33746] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@160c5540, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:41,380 INFO [RS:1;asf911:43833] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x4bffc363 to localhost:59094 2018-01-13 07:25:41,381 DEBUG [RS:1;asf911:43833] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,381 INFO [RS:1;asf911:43833] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:41,381 INFO [RS:1;asf911:43833] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:41,381 INFO [RS:1;asf911:43833] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:41,380 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:41,381 INFO [M:0;asf911:33746] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:41,383 DEBUG [M:0;asf911:33746] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:41,382 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1388): Waiting on 1 regions to close 2018-01-13 07:25:41,383 DEBUG [RS_CLOSE_META-asf911:43833-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:25:41,383 INFO [M:0;asf911:33746] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x100d5392 to localhost:59094 2018-01-13 07:25:41,383 DEBUG [RS:1;asf911:43833] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-01-13 07:25:41,384 DEBUG [M:0;asf911:33746] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,384 DEBUG [RS_CLOSE_META-asf911:43833-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:41,386 DEBUG [RS_CLOSE_META-asf911:43833-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:41,387 INFO [M:0;asf911:33746] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,33746,1515828335861; all regions closed. 2018-01-13 07:25:41,387 DEBUG [M:0;asf911:33746] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,387 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:41,387 INFO [M:0;asf911:33746] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,33746,1515828335861 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:33746 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,33746,1515828335861-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,33746,1515828335861-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,33746,1515828335861-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,33746,1515828335861-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,33746,1515828335861-MobCompactionChore Period: 604800 Unit: SECONDS]] on shutdown 2018-01-13 07:25:41,387 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:41,388 INFO [M:0;asf911:33746] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:25:41,388 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:41,388 INFO [M:0;asf911:33746] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:25:41,388 DEBUG [M:0;asf911:33746] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:41,388 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:41,388 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:41,389 ERROR [RS_CLOSE_META-asf911:43833-0] regionserver.HRegion(1620): Memstore size is 1330 2018-01-13 07:25:41,389 DEBUG [RS_CLOSE_META-asf911:43833-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:41,389 INFO [RS_CLOSE_META-asf911:43833-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:41,389 DEBUG [RS_CLOSE_META-asf911:43833-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:41,398 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:41,398 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:41,398 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:41,398 DEBUG [M:0;asf911:33746] zookeeper.ZKUtil(614): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:41,400 WARN [M:0;asf911:33746] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:41,400 INFO [M:0;asf911:33746] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:25:41,398 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:41,400 INFO [M:0;asf911:33746] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x73de3603 to localhost:59094 2018-01-13 07:25:41,400 DEBUG [M:0;asf911:33746] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,400 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:41,400 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:41,398 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:25:41,401 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,42367,1515828335955' ***** 2018-01-13 07:25:41,402 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:25:41,401 INFO [M:0;asf911:33746] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:25:41,400 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:41,400 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:41,402 INFO [M:0;asf911:33746] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:25:41,404 INFO [M:0;asf911:33746] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:25:41,404 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,406 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,406 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,406 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,409 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,409 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,412 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,411 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,411 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,411 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,411 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:41,425 DEBUG [main-EventThread] master.ActiveMasterManager(132): No master available. Notifying waiting threads 2018-01-13 07:25:41,411 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:25:41,428 INFO [M:1;asf911:42367] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x34e3c871 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:41,428 DEBUG [main] zookeeper.ZKUtil(614): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:41,429 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:41,429 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:25:41,429 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:25:41,430 DEBUG [main] zookeeper.ZKUtil(614): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:41,430 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:25:41,430 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:41,430 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,33746,1515828335861 2018-01-13 07:25:41,431 INFO [M:0;asf911:33746] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,33746,1515828335861_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:41,432 INFO [M:0;asf911:33746] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:25:41,432 INFO [M:0;asf911:33746] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:41,448 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:41,448 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:41,448 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:41,449 DEBUG [M:1;asf911:42367] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3a24209d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:41,448 DEBUG [M:0;asf911:33746] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,33746,1515828335861 already deleted, retry=false 2018-01-13 07:25:41,448 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:41,449 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:41,449 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:41,449 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:41,456 INFO [M:0;asf911:33746] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,33746,1515828335861; zookeeper connection closed. 2018-01-13 07:25:41,457 INFO [M:0;asf911:33746] regionserver.HRegionServer(1157): M:0;asf911:33746 exiting 2018-01-13 07:25:41,459 WARN [main-EventThread] zookeeper.ZKUtil(361): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Unable to set watcher on znode /hbase/running org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:25:41,459 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:33746-0x160ee689450001f, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:25:41,572 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,53530,1515828336015; all regions closed. 2018-01-13 07:25:41,578 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:41,579 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:41,580 DEBUG [RS:0;asf911:53530] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,581 INFO [RS:0;asf911:53530] regionserver.Leases(147): RS:0;asf911:53530 closing leases 2018-01-13 07:25:41,581 INFO [RS:0;asf911:53530] regionserver.Leases(150): RS:0;asf911:53530 closed leases 2018-01-13 07:25:41,581 INFO [RS:0;asf911:53530] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,53530,1515828336015 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,53530,1515828336015 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:41,581 INFO [RS:0;asf911:53530] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:41,581 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:41,581 INFO [RS:0;asf911:53530] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:41,582 INFO [RS:0;asf911:53530] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:41,582 INFO [RS:0;asf911:53530] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:41,585 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,43833,1515828336110; all regions closed. 2018-01-13 07:25:41,590 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:41,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:41,590 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:41,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:53530-0x160ee6894500021, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:41,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,53530,1515828336015 2018-01-13 07:25:41,592 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:41,596 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741851_1027{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:41,597 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741851_1027{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:41,598 DEBUG [RS:1;asf911:43833] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:41,598 INFO [RS:1;asf911:43833] regionserver.Leases(147): RS:1;asf911:43833 closing leases 2018-01-13 07:25:41,598 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,53530,1515828336015; zookeeper connection closed. 2018-01-13 07:25:41,599 INFO [RS:0;asf911:53530] regionserver.HRegionServer(1157): RS:0;asf911:53530 exiting 2018-01-13 07:25:41,599 INFO [RS:1;asf911:43833] regionserver.Leases(150): RS:1;asf911:43833 closed leases 2018-01-13 07:25:41,599 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@57f60459] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@57f60459 2018-01-13 07:25:41,600 INFO [RS:1;asf911:43833] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,43833,1515828336110 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,43833,1515828336110 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:41,600 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:41,600 INFO [RS:1;asf911:43833] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:41,614 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:43833-0x160ee6894500022, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,43833,1515828336110 2018-01-13 07:25:41,623 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,43833,1515828336110; zookeeper connection closed. 2018-01-13 07:25:41,624 INFO [RS:1;asf911:43833] regionserver.HRegionServer(1157): RS:1;asf911:43833 exiting 2018-01-13 07:25:41,625 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@37efd508] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@37efd508 2018-01-13 07:25:41,625 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:25:41,737 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:41,738 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:41,738 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:41,738 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:42,451 INFO [M:1;asf911:42367] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:42,452 INFO [M:1;asf911:42367] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,42367,1515828335955 2018-01-13 07:25:42,452 DEBUG [M:1;asf911:42367] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:42,452 INFO [M:1;asf911:42367] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x34e3c871 to localhost:59094 2018-01-13 07:25:42,452 DEBUG [M:1;asf911:42367] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:42,453 INFO [M:1;asf911:42367] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,42367,1515828335955; all regions closed. 2018-01-13 07:25:42,453 INFO [M:1;asf911:42367] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,42367,1515828335955 had [] on shutdown 2018-01-13 07:25:42,453 DEBUG [M:1;asf911:42367] master.HMaster(1187): Stopping service threads 2018-01-13 07:25:42,457 DEBUG [M:1;asf911:42367] zookeeper.ZKUtil(614): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:25:42,457 WARN [M:1;asf911:42367] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:25:42,465 DEBUG [M:1;asf911:42367] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,42367,1515828335955 already deleted, retry=false 2018-01-13 07:25:42,473 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42367-0x160ee6894500020, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,42367,1515828335955 2018-01-13 07:25:42,473 INFO [M:1;asf911:42367] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,42367,1515828335955; zookeeper connection closed. 2018-01-13 07:25:42,473 INFO [M:1;asf911:42367] regionserver.HRegionServer(1157): M:1;asf911:42367 exiting 2018-01-13 07:25:42,504 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:42,507 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741849_1025 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,507 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741851_1027 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:42,507 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741852_1028 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741850_1026 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741848_1024 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741847_1023 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741853_1029 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741854_1030 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,508 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741846_1022 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,509 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741845_1021 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:42,509 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:42,523 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:42,525 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee689450002c connected 2018-01-13 07:25:42,525 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:25:42,526 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:25:42,527 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:25:42,527 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:25:42,528 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:25:42,529 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:25:42,529 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:25:42,530 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:25:42,530 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:25:42,531 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:25:42,532 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:25:42,533 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:25:42,533 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:25:42,537 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:25:42,537 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:25:42,538 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:25:42,539 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:25:42,540 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:25:42,541 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:25:42,543 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:25:42,544 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:42,557 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:25:42,558 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:25:42,559 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:25:42,564 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:42,565 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:42,565 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:42,565 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee689450002c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:25:42,601 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testZNodeDeletes Thread=283 (was 225) Potentially hanging thread: RS-EventLoopGroup-15-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-15-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:33746-HFileCleaner.small.0-1515828336510 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-15-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:33746-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: M:0;asf911:33746-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-15-24 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:33746-HFileCleaner.large.0-1515828336510 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-15-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-23 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1624177010_1 at /127.0.0.1:54493 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2059 (was 1651) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=199 (was 208), ProcessCount=270 (was 271), AvailableMemoryMB=10842 (was 10979) 2018-01-13 07:25:42,602 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2059 is superior to 1024 2018-01-13 07:25:42,632 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testRegionServerSessionExpired Thread=281, OpenFileDescriptor=2055, MaxFileDescriptor=60000, SystemLoadAverage=199, ProcessCount=270, AvailableMemoryMB=10829 2018-01-13 07:25:42,632 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2055 is superior to 1024 2018-01-13 07:25:42,652 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741855_1031{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 7 2018-01-13 07:25:42,653 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741855_1031 size 7 2018-01-13 07:25:43,056 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:25:43,056 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:25:43,059 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:43,060 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:43,060 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:43,060 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:43,060 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:43,062 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:42210 2018-01-13 07:25:43,062 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,063 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,064 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,065 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,067 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:42210 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:43,074 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:422100x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:43,075 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:42210-0x160ee689450002d connected 2018-01-13 07:25:43,141 DEBUG [main] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:43,141 DEBUG [main] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:43,142 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=42210 2018-01-13 07:25:43,143 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=42210 2018-01-13 07:25:43,144 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=42210 2018-01-13 07:25:43,145 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:43,147 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:43,147 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:43,148 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:43,148 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:43,148 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:43,149 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:50456 2018-01-13 07:25:43,149 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,150 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,151 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,152 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,154 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:50456 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:43,165 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:504560x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:43,166 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:50456-0x160ee689450002e connected 2018-01-13 07:25:43,173 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:25:43,178 DEBUG [main] zookeeper.ZKUtil(357): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:43,179 DEBUG [main] zookeeper.ZKUtil(357): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:43,179 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=50456 2018-01-13 07:25:43,180 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=50456 2018-01-13 07:25:43,181 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=50456 2018-01-13 07:25:43,182 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:25:43,206 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:43,206 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:43,207 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:43,207 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:43,207 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:43,207 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:43,208 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:59071 2018-01-13 07:25:43,208 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,209 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,210 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,211 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,212 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:59071 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:43,224 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:590710x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:43,225 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:59071-0x160ee689450002f connected 2018-01-13 07:25:43,225 DEBUG [main] zookeeper.ZKUtil(357): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:43,226 DEBUG [main] zookeeper.ZKUtil(357): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:43,226 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=59071 2018-01-13 07:25:43,228 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=59071 2018-01-13 07:25:43,229 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=59071 2018-01-13 07:25:43,255 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:25:43,255 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:25:43,255 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:25:43,255 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:25:43,256 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:25:43,256 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:25:43,257 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:58799 2018-01-13 07:25:43,258 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,258 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,259 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,261 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,262 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:58799 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:43,273 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:587990x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:43,275 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:58799-0x160ee6894500030 connected 2018-01-13 07:25:43,275 DEBUG [main] zookeeper.ZKUtil(357): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:25:43,276 DEBUG [main] zookeeper.ZKUtil(357): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:25:43,276 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=58799 2018-01-13 07:25:43,278 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=58799 2018-01-13 07:25:43,279 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=58799 2018-01-13 07:25:43,281 INFO [M:0;asf911:42210] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,281 INFO [M:1;asf911:50456] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,50456,1515828343146 2018-01-13 07:25:43,290 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,291 DEBUG [M:1;asf911:50456] zookeeper.ZKUtil(355): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,50456,1515828343146 2018-01-13 07:25:43,298 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:43,298 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:43,298 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:43,298 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:25:43,307 DEBUG [M:1;asf911:50456] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:25:43,307 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:43,310 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:43,310 INFO [M:0;asf911:42210] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,42210,1515828343058 from backup master directory 2018-01-13 07:25:43,311 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:43,311 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:43,310 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:25:43,310 DEBUG [M:1;asf911:50456] zookeeper.ZKUtil(355): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:25:43,315 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,315 WARN [M:0;asf911:42210] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:43,315 INFO [M:1;asf911:50456] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,42210,1515828343058; waiting to become the next active master 2018-01-13 07:25:43,315 INFO [M:0;asf911:42210] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,340 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:43,341 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:43,343 DEBUG [M:0;asf911:42210] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 877180de-c7f4-4e2b-9790-6b57c6c3a409 2018-01-13 07:25:43,349 INFO [M:0;asf911:42210] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:25:43,350 INFO [M:0;asf911:42210] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:25:43,365 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:43,365 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741857_1033 size 32 2018-01-13 07:25:43,366 DEBUG [M:0;asf911:42210] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:43,371 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:43,372 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:43,372 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,372 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:43,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:43,376 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:43,377 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:43,381 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:43,381 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:43,382 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,382 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:43,386 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:43,387 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:43,387 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,387 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:43,391 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:43,391 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:43,392 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:43,392 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:43,394 DEBUG [M:0;asf911:42210] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:43,397 DEBUG [M:0;asf911:42210] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:43,402 DEBUG [M:0;asf911:42210] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:43,402 INFO [M:0;asf911:42210] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:25:43,402 DEBUG [M:0;asf911:42210] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:43,402 DEBUG [M:0;asf911:42210] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:43,403 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:43,403 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:43,403 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:43,403 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:43,403 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:43,404 INFO [M:0;asf911:42210] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:43,419 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:43,419 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:43,423 DEBUG [M:0;asf911:42210] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:43,443 INFO [M:0;asf911:42210] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:25:43,444 INFO [M:0;asf911:42210] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:25:43,444 DEBUG [M:0;asf911:42210] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:25:43,458 INFO [M:0;asf911:42210] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x0b6462ae to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:43,467 DEBUG [M:0;asf911:42210] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5407c994, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:43,489 INFO [M:0;asf911:42210] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:25:43,490 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:25:43,491 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:25:43,499 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:25:43,499 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:25:43,500 INFO [M:0;asf911:42210] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:25:43,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:43,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:43,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:43,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:25:43,507 INFO [M:0;asf911:42210] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,42210,1515828343058, sessionid=0x160ee689450002d, setting cluster-up flag (Was=false) 2018-01-13 07:25:43,526 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:25:43,548 INFO [M:0;asf911:42210] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:25:43,550 DEBUG [M:0;asf911:42210] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,612 INFO [M:0;asf911:42210] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:25:43,614 DEBUG [M:0;asf911:42210] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:43,616 INFO [M:0;asf911:42210] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:25:43,616 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:42210, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:43,617 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:42210, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:43,617 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:42210, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:43,617 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:42210, corePoolSize=5, maxPoolSize=5 2018-01-13 07:25:43,617 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:42210, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:43,617 DEBUG [M:0;asf911:42210] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:42210, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:43,620 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:25:43,620 INFO [M:0;asf911:42210] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:25:43,623 DEBUG [M:0;asf911:42210] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:25:43,625 INFO [M:0;asf911:42210] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:25:43,625 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 5msec 2018-01-13 07:25:43,625 DEBUG [M:0;asf911:42210] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:25:43,625 DEBUG [M:0;asf911:42210] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:25:43,625 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:25:43,628 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828373628 2018-01-13 07:25:43,628 INFO [M:0;asf911:42210] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:25:43,628 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:25:43,629 INFO [M:0;asf911:42210] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:25:43,629 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:25:43,629 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:25:43,629 INFO [M:0;asf911:42210] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:25:43,631 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:25:43,631 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:25:43,632 DEBUG [M:0;asf911:42210] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:25:43,632 DEBUG [M:0;asf911:42210] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:42210-HFileCleaner.large.0-1515828343632,5,main] 2018-01-13 07:25:43,632 DEBUG [M:0;asf911:42210] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:42210-HFileCleaner.small.0-1515828343632,5,main] 2018-01-13 07:25:43,632 INFO [M:0;asf911:42210] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828343290, completionTime=-1 2018-01-13 07:25:43,632 INFO [M:0;asf911:42210] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:25:43,640 DEBUG [M:0;asf911:42210-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:25:43,641 DEBUG [M:0;asf911:42210-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500032 connected 2018-01-13 07:25:43,686 INFO [RS:0;asf911:59071] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x62cb62e5 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:43,688 INFO [RS:1;asf911:58799] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x70b71f65 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:43,707 DEBUG [RS:0;asf911:59071] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c0c90cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:43,709 DEBUG [RS:1;asf911:58799] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e70ff03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:43,709 INFO [RS:0;asf911:59071] regionserver.HRegionServer(866): ClusterId : 877180de-c7f4-4e2b-9790-6b57c6c3a409 2018-01-13 07:25:43,709 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:43,715 DEBUG [RS:0;asf911:59071] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:43,715 INFO [RS:1;asf911:58799] regionserver.HRegionServer(866): ClusterId : 877180de-c7f4-4e2b-9790-6b57c6c3a409 2018-01-13 07:25:43,715 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:25:43,723 DEBUG [RS:1;asf911:58799] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:25:43,723 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:43,724 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:43,732 DEBUG [RS:0;asf911:59071] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:43,732 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:25:43,732 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:25:43,740 DEBUG [RS:1;asf911:58799] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:25:43,740 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:43,740 DEBUG [RS:0;asf911:59071] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68330995, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:43,741 DEBUG [RS:0;asf911:59071] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:59071 2018-01-13 07:25:43,741 INFO [RS:0;asf911:59071] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:43,741 INFO [RS:0;asf911:59071] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:43,741 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:25:43,741 DEBUG [RS:1;asf911:58799] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4f7415, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:25:43,742 INFO [RS:0;asf911:59071] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,42210,1515828343058 with port=59071, startcode=1515828343205 2018-01-13 07:25:43,742 DEBUG [RS:1;asf911:58799] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:58799 2018-01-13 07:25:43,742 INFO [RS:1;asf911:58799] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:25:43,742 INFO [RS:1;asf911:58799] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:25:43,742 DEBUG [RS:0;asf911:59071] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:25:43,743 INFO [RS:1;asf911:58799] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,42210,1515828343058 with port=58799, startcode=1515828343253 2018-01-13 07:25:43,743 DEBUG [RS:1;asf911:58799] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:25:43,745 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.6 (auth:SIMPLE) 2018-01-13 07:25:43,745 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.7 (auth:SIMPLE) 2018-01-13 07:25:43,745 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51921 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:43,745 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 47990 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:43,750 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42210] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,752 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,753 DEBUG [RS:0;asf911:59071] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:43,754 DEBUG [RS:0;asf911:59071] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:43,754 DEBUG [RS:0;asf911:59071] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:43,755 DEBUG [RS:1;asf911:58799] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:25:43,756 DEBUG [RS:1;asf911:58799] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:25:43,756 DEBUG [RS:1;asf911:58799] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:25:43,765 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:43,773 DEBUG [RS:0;asf911:59071] zookeeper.ZKUtil(355): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,774 WARN [RS:0;asf911:59071] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:43,774 DEBUG [RS:1;asf911:58799] zookeeper.ZKUtil(355): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,774 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,774 WARN [RS:1;asf911:58799] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:25:43,774 DEBUG [RS:0;asf911:59071] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,774 DEBUG [RS:1;asf911:58799] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,775 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,781 INFO [RS:1;asf911:58799] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:43,781 INFO [RS:0;asf911:59071] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:43,783 DEBUG [RS:1;asf911:58799] zookeeper.ZKUtil(355): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,783 DEBUG [RS:0;asf911:59071] zookeeper.ZKUtil(355): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,783 DEBUG [RS:1;asf911:58799] zookeeper.ZKUtil(355): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,783 DEBUG [RS:0;asf911:59071] zookeeper.ZKUtil(355): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,785 INFO [M:0;asf911:42210] master.ServerManager(891): Finished wait on RegionServer count=2; waited=153ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:25:43,785 INFO [RS:1;asf911:58799] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x0fd6ca80 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:43,785 INFO [RS:0;asf911:59071] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x6cd9acf6 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:43,786 INFO [M:0;asf911:42210] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:25:43,799 DEBUG [RS:1;asf911:58799] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f620a77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:43,799 DEBUG [RS:1;asf911:58799] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:43,799 DEBUG [RS:0;asf911:59071] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d1e65b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:43,800 INFO [RS:1;asf911:58799] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:43,800 DEBUG [RS:0;asf911:59071] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:25:43,800 INFO [RS:0;asf911:59071] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:25:43,804 INFO [RS:1;asf911:58799] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:43,805 INFO [RS:0;asf911:59071] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:25:43,805 INFO [RS:1;asf911:58799] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:43,805 INFO [RS:0;asf911:59071] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:25:43,805 INFO [RS:1;asf911:58799] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:43,806 INFO [RS:0;asf911:59071] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:25:43,810 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:58799, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,810 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:59071, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,810 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:58799, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:43,810 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:59071, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:43,811 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:58799, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,811 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:59071, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,811 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:58799, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,811 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:59071, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,811 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:58799, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:43,811 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:59071, corePoolSize=1, maxPoolSize=1 2018-01-13 07:25:43,811 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:58799, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:43,812 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:59071, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:43,812 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:58799, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:43,812 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:59071, corePoolSize=10, maxPoolSize=10 2018-01-13 07:25:43,812 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:58799, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,812 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:59071, corePoolSize=3, maxPoolSize=3 2018-01-13 07:25:43,812 DEBUG [RS:1;asf911:58799] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:58799, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:43,812 DEBUG [RS:0;asf911:59071] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:59071, corePoolSize=2, maxPoolSize=2 2018-01-13 07:25:43,824 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741845_1021, blk_1073741846_1022, blk_1073741847_1023, blk_1073741848_1024, blk_1073741849_1025, blk_1073741850_1026, blk_1073741851_1027, blk_1073741852_1028, blk_1073741853_1029, blk_1073741854_1030] 2018-01-13 07:25:43,837 INFO [RS:1;asf911:58799] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:43,837 INFO [SplitLogWorker-asf911:58799] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,58799,1515828343253 starting 2018-01-13 07:25:43,842 INFO [RS:0;asf911:59071] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:25:43,842 INFO [SplitLogWorker-asf911:59071] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,59071,1515828343205 starting 2018-01-13 07:25:43,857 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,58799,1515828343253, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:58799, sessionid=0x160ee6894500030 2018-01-13 07:25:43,857 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:43,858 DEBUG [RS:1;asf911:58799] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,858 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,58799,1515828343253' 2018-01-13 07:25:43,858 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:43,858 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:43,859 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:43,859 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:43,859 DEBUG [RS:1;asf911:58799] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:43,859 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,58799,1515828343253' 2018-01-13 07:25:43,859 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:43,860 DEBUG [RS:1;asf911:58799] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:43,860 DEBUG [RS:1;asf911:58799] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:43,861 INFO [RS:1;asf911:58799] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:43,861 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,59071,1515828343205, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:59071, sessionid=0x160ee689450002f 2018-01-13 07:25:43,861 INFO [RS:1;asf911:58799] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:43,861 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:25:43,861 DEBUG [RS:0;asf911:59071] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,861 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,59071,1515828343205' 2018-01-13 07:25:43,861 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:25:43,861 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:25:43,862 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:25:43,862 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:25:43,862 DEBUG [RS:0;asf911:59071] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:43,862 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,59071,1515828343205' 2018-01-13 07:25:43,862 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:25:43,863 DEBUG [RS:0;asf911:59071] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:25:43,863 DEBUG [RS:0;asf911:59071] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:25:43,863 INFO [RS:0;asf911:59071] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:25:43,863 INFO [RS:0;asf911:59071] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:25:43,961 DEBUG [M:0;asf911:42210] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:43,962 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:43,962 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:43,963 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:43,966 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:25:44,004 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:25:44,005 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:25:44,005 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:25:44,005 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:25:44,045 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:25:44,046 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:25:44,196 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:25:44,197 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:25:44,205 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:44,867 INFO [RS:1;asf911:58799] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C58799%2C1515828343253, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:44,869 INFO [RS:0;asf911:59071] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C59071%2C1515828343205, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:44,879 DEBUG [RS-EventLoopGroup-19-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:25:44,879 DEBUG [RS-EventLoopGroup-19-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:44,880 DEBUG [RS-EventLoopGroup-19-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:44,880 DEBUG [RS-EventLoopGroup-19-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:25:44,885 INFO [RS:0;asf911:59071] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869 2018-01-13 07:25:44,885 INFO [RS:1;asf911:58799] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253/asf911.gq1.ygridcore.net%2C58799%2C1515828343253.1515828344868 2018-01-13 07:25:44,885 DEBUG [RS:0;asf911:59071] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:25:44,885 DEBUG [RS:1;asf911:58799] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:25:44,888 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:44,898 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:25:44,907 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,059 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,59071,1515828343205 version=3145728 2018-01-13 07:25:45,060 DEBUG [RSProcedureDispatcher-pool29-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,060 DEBUG [RSProcedureDispatcher-pool29-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:45,064 INFO [RS-EventLoopGroup-18-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:45,064 INFO [RS-EventLoopGroup-18-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34003 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:45,066 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59071] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:25:45,067 INFO [RS_OPEN_META-asf911:59071-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:45,071 INFO [RS_OPEN_META-asf911:59071-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:45,078 DEBUG [RS-EventLoopGroup-19-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:25:45,079 DEBUG [RS-EventLoopGroup-19-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:45,084 INFO [RS_OPEN_META-asf911:59071-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:45,085 DEBUG [RS_OPEN_META-asf911:59071-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:25:45,085 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:45,085 DEBUG [RS_OPEN_META-asf911:59071-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:25:45,085 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:25:45,086 INFO [RS_OPEN_META-asf911:59071-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:25:45,086 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:25:45,086 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:45,091 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:45,092 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:45,092 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:45,092 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:45,093 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:45,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:45,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:45,096 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:45,096 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:45,097 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:45,100 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:45,100 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:45,100 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:45,100 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:45,101 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:45,104 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:45,104 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:45,104 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:45,104 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:45,105 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:45,107 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:45,107 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:45,107 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:45,108 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:45,108 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:45,112 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:45,115 DEBUG [RS_OPEN_META-asf911:59071-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:45,121 DEBUG [RS_OPEN_META-asf911:59071-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:25:45,122 INFO [RS_OPEN_META-asf911:59071-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:25:45,125 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:25:45,131 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=42210] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,132 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:25:45,132 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828345131}]},"ts":1515828345131} 2018-01-13 07:25:45,134 DEBUG [RS_OPEN_META-asf911:59071-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,138 DEBUG [hconnection-0x2ec470b9-shared-pool32-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:45,140 INFO [RS-EventLoopGroup-18-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:45,141 INFO [RS-EventLoopGroup-18-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34006 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:45,144 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:25:45,146 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,157 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:25:45,198 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:25:45,239 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:25:45,239 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.1520sec 2018-01-13 07:25:45,281 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.4530sec 2018-01-13 07:25:45,282 INFO [M:0;asf911:42210] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,292 INFO [M:0;asf911:42210] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253 belongs to an existing region server 2018-01-13 07:25:45,293 INFO [M:0;asf911:42210] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205 belongs to an existing region server 2018-01-13 07:25:45,293 INFO [M:0;asf911:42210] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:25:45,296 INFO [M:0;asf911:42210] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:25:45,296 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828350296 2018-01-13 07:25:45,296 INFO [M:0;asf911:42210] assignment.AssignmentManager(1219): Joined the cluster in 3msec, failover=true 2018-01-13 07:25:45,301 INFO [M:0;asf911:42210] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:25:45,302 INFO [M:0;asf911:42210] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:25:45,449 DEBUG [M:0;asf911:42210] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:45,450 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:45,498 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:45,516 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:45,516 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:45,520 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:45,522 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:25:45,539 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:45,539 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741864_1040 size 42 2018-01-13 07:25:45,540 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:45,540 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.: disabling compactions & flushes 2018-01-13 07:25:45,540 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:45,540 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:45,602 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:45,604 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:25:45,610 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:25:45,654 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:45,654 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828345654}]},"ts":1515828345654} 2018-01-13 07:25:45,658 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:25:45,658 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:45,665 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205}] 2018-01-13 07:25:45,712 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205 hbase:namespace hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:45,715 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59071,1515828343205; forceNewPlan=false, retain=false 2018-01-13 07:25:45,868 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:25:45,871 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:45,872 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x528738c4 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:25:45,890 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c40cce4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:25:45,894 DEBUG [MultiHConnection-shared--pool33-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:45,896 INFO [RS-EventLoopGroup-18-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:45,897 INFO [RS-EventLoopGroup-18-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34015 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:45,900 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205; rit=OPENING, location=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:46,051 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,59071,1515828343205 version=3145728 2018-01-13 07:25:46,054 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=59071] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:46,060 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] regionserver.HRegion(6949): Opening region: {ENCODED => 408ed851e839a1b5b1051712e18f98f9, NAME => 'hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:46,061 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 408ed851e839a1b5b1051712e18f98f9 2018-01-13 07:25:46,062 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:46,067 DEBUG [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info 2018-01-13 07:25:46,068 DEBUG [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info 2018-01-13 07:25:46,068 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:46,068 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:46,068 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:46,071 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9 2018-01-13 07:25:46,078 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:46,078 INFO [RS_OPEN_PRIORITY_REGION-asf911:59071-0] regionserver.HRegion(960): Onlined 408ed851e839a1b5b1051712e18f98f9; next sequenceid=2 2018-01-13 07:25:46,080 INFO [PostOpenDeployTasks:408ed851e839a1b5b1051712e18f98f9] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:46,083 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=42210] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205; rit=OPENING, location=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:46,083 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:46,083 DEBUG [PostOpenDeployTasks:408ed851e839a1b5b1051712e18f98f9] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:46,086 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:59071-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. on asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:46,147 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:46,189 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:25:46,189 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:46,189 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9, target=asf911.gq1.ygridcore.net,59071,1515828343205 in 429msec 2018-01-13 07:25:46,190 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828346190}]},"ts":1515828346190} 2018-01-13 07:25:46,195 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:25:46,264 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:25:46,269 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:25:46,281 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:46,316 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 967msec 2018-01-13 07:25:46,357 DEBUG [M:0;asf911:42210] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,359 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,399 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,440 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,491 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:46,557 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:25:46,599 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 269msec 2018-01-13 07:25:46,749 DEBUG [M:0;asf911:42210] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:46,751 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:46,799 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:46,824 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741845_1021, blk_1073741846_1022, blk_1073741847_1023, blk_1073741848_1024, blk_1073741849_1025, blk_1073741850_1026, blk_1073741851_1027, blk_1073741852_1028, blk_1073741853_1029, blk_1073741854_1030] 2018-01-13 07:25:46,844 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:46,893 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:46,907 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:25:46,960 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:25:47,001 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 358msec 2018-01-13 07:25:47,022 DEBUG [M:0;asf911:42210] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:25:47,231 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:25:47,255 DEBUG [M:0;asf911:42210] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:25:47,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:25:47,265 INFO [M:0;asf911:42210] master.HMaster(928): Master has completed initialization 3.948sec 2018-01-13 07:25:47,266 INFO [M:0;asf911:42210] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:25:47,266 INFO [M:0;asf911:42210] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:25:47,268 INFO [M:0;asf911:42210] regionserver.HRegionServer(866): ClusterId : 877180de-c7f4-4e2b-9790-6b57c6c3a409 2018-01-13 07:25:47,290 DEBUG [hconnection-0x51a18b21-shared-pool12-t4] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:47,292 DEBUG [Default-IPC-NioEventLoopGroup-7-6] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:43833 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:43833 2018-01-13 07:25:47,397 DEBUG [hconnection-0x51a18b21-shared-pool12-t4] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:47,399 INFO [RS-EventLoopGroup-18-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:47,400 INFO [RS-EventLoopGroup-18-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34020 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:47,404 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:47,405 INFO [Time-limited test] hbase.TestZooKeeper(128): Starting testRegionServerSessionExpired 2018-01-13 07:25:47,410 INFO [Time-limited test-EventThread] hbase.HBaseTestingUtility$2(2686): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2018-01-13 07:25:47,422 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:47,422 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:47,423 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,59071,1515828343205] 2018-01-13 07:25:47,422 INFO [Time-limited test] hbase.HBaseTestingUtility(2702): ZK Closed Session 0x160ee689450002f 2018-01-13 07:25:47,425 INFO [main-EventThread] master.ServerManager(600): Processing expiration of asf911.gq1.ygridcore.net,59071,1515828343205 on asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:25:47,426 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:47,428 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 znode expired, triggering replicatorRemoved event 2018-01-13 07:25:47,429 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:47,430 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:47,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-01-13 07:25:47,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(528): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-01-13 07:25:47,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-01-13 07:25:47,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(528): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-01-13 07:25:47,510 INFO [Time-limited test-EventThread] hbase.HBaseTestingUtility$2(2686): Monitor ZKW received event=WatchedEvent state:Disconnected type:None path:null 2018-01-13 07:25:47,512 INFO [Time-limited test] hbase.TestZooKeeper(165): Creating table testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:47,526 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:25:47,528 INFO [RS-EventLoopGroup-16-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:47,529 INFO [RS-EventLoopGroup-16-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 46047 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:47,548 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.HMaster$3(1743): Client=jenkins//67.195.81.155 create 'testRegionServerSessionExpired_1515828347510', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-13 07:25:47,609 DEBUG [main-EventThread] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:47,609 DEBUG [main-EventThread] assignment.AssignmentManager(1393): Added=asf911.gq1.ygridcore.net,59071,1515828343205 to dead servers, submitted shutdown handler to be executed meta=true 2018-01-13 07:25:47,611 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:25:47,611 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:47,612 INFO [ProcExecWrkr-14] procedure.ServerCrashProcedure(118): Start pid=7, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:47,612 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] procedure2.ProcedureExecutor(869): Stored pid=8, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:47,756 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_PROCESS_META; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:47,757 DEBUG [ProcExecWrkr-15] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:47,757 DEBUG [ProcExecWrkr-14] procedure.ServerCrashProcedure(192): pid=7, state=RUNNABLE:SERVER_CRASH_PROCESS_META; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true; Processing hbase:meta that was on asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:47,758 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=9, ppid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true}] 2018-01-13 07:25:47,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:47,813 DEBUG [ProcExecWrkr-15] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:47,813 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=9, ppid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true 2018-01-13 07:25:47,813 INFO [ProcExecWrkr-14] procedure.RecoverMetaProcedure(106): Start pid=9, ppid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true 2018-01-13 07:25:47,816 DEBUG [ProcExecWrkr-14] master.MasterWalManager(257): Renamed region directory: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting 2018-01-13 07:25:47,816 INFO [ProcExecWrkr-14] master.SplitLogManager(455): dead splitlog workers [asf911.gq1.ygridcore.net,59071,1515828343205] 2018-01-13 07:25:47,820 INFO [ProcExecWrkr-14] master.SplitLogManager(235): Started splitting 1 logs in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting] for [asf911.gq1.ygridcore.net,59071,1515828343205] 2018-01-13 07:25:47,829 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:47,829 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:47,832 DEBUG [ProcExecWrkr-15] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/default/testRegionServerSessionExpired_1515828347510/.tabledesc/.tableinfo.0000000001 2018-01-13 07:25:47,834 INFO [RegionOpenAndInitThread-testRegionServerSessionExpired_1515828347510-1] regionserver.HRegion(6772): creating HRegion testRegionServerSessionExpired_1515828347510 HTD == 'testRegionServerSessionExpired_1515828347510', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:47,847 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:47,848 DEBUG [SplitLogWorker-asf911:58799] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,58799,1515828343253 is ready to take more tasks, will get task list and try grab tasks again. 2018-01-13 07:25:47,848 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741866_1042{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 79 2018-01-13 07:25:47,849 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(331): put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:47,849 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741866_1042 size 79 2018-01-13 07:25:47,856 INFO [SplitLogWorker-asf911:58799] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,58799,1515828343253 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:47,858 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(453): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta acquired by asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:47,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:47,888 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter(251): Splitting WAL=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta, length=0 2018-01-13 07:25:47,898 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:47,898 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:47,898 WARN [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter(710): File hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta might be still open, length is 0 2018-01-13 07:25:47,899 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.ServerManager(394): Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server 2018-01-13 07:25:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] ipc.CallRunner(141): callId: 6 service: RegionServerStatusService methodName: RegionServerReport size: 449 connection: 67.195.81.155:51921 deadline: 1515828357901 org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:25:47,905 INFO [IPC Server handler 2 on 45316] blockmanagement.BlockInfoContiguousUnderConstruction(331): BLOCK* blk_1073741862_1038{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} recovery started, primary=ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW] 2018-01-13 07:25:47,905 WARN [IPC Server handler 2 on 45316] namenode.FSNamesystem(4102): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta has not been closed. Lease recovery is in progress. RecoveryId = 1043 for block blk_1073741862_1038{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} 2018-01-13 07:25:47,911 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] util.FSHDFSUtils(284): Failed to recover lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta after 12ms 2018-01-13 07:25:47,937 ERROR [RS:0;asf911:59071] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,59071,1515828343205: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) ***** org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:93) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:83) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:363) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:340) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1203) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException): org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-01-13 07:25:47,939 ERROR [RS:0;asf911:59071] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:47,944 INFO [RS:0;asf911:59071] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 502701952 }, "NonHeapMemoryUsage" : { "committed" : 119996416, "init" : 2555904, "max" : -1, "used" : 118190800 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 17222, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 87, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 7, "TotalCallTime_99th_percentile" : 7, "TotalCallTime_99.9th_percentile" : 7, "TotalCallTime_TimeRangeCount_0-1" : 17, "TotalCallTime_TimeRangeCount_1-3" : 9, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 58, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 87, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 55, "ResponseSize_90th_percentile" : 114, "ResponseSize_95th_percentile" : 768, "ResponseSize_98th_percentile" : 864, "ResponseSize_99th_percentile" : 868, "ResponseSize_99.9th_percentile" : 872, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_100000000-inf" : 58, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 87, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 4, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 18, "ProcessCallTime_TimeRangeCount_1-3" : 8, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 14, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 87, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 86, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 130, "RequestSize_90th_percentile" : 377, "RequestSize_95th_percentile" : 517, "RequestSize_98th_percentile" : 734, "RequestSize_99th_percentile" : 735, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 17, "RequestSize_SizeRangeCount_100-1000" : 12, "RequestSize_SizeRangeCount_100000000-inf" : 58, "sentBytes" : 7925, "QueueCallTime_num_ops" : 87, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "QueueCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:47,946 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,59071,1515828343205 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,59071,1515828343205: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) ***** Cause: org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:93) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:83) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:363) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:340) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1203) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.YouAreDeadException): org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-01-13 07:25:47,947 INFO [RS:0;asf911:59071] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,59071,1515828343205' ***** 2018-01-13 07:25:47,947 INFO [RS:0;asf911:59071] regionserver.HRegionServer(2172): STOPPED: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing asf911.gq1.ygridcore.net,59071,1515828343205 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:395) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:300) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:456) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13088) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:25:47,947 INFO [RS:0;asf911:59071] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:25:47,947 INFO [RS:0;asf911:59071] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:25:47,947 INFO [SplitLogWorker-asf911:59071] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:25:47,947 INFO [RS:0;asf911:59071] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:25:47,948 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:25:47,947 INFO [SplitLogWorker-asf911:59071] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,59071,1515828343205 exiting 2018-01-13 07:25:47,948 INFO [RS:0;asf911:59071] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:25:47,947 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:25:47,949 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:47,949 DEBUG [RS:0;asf911:59071] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:25:47,949 DEBUG [RS_CLOSE_REGION-asf911:59071-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:47,950 INFO [RS:0;asf911:59071] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x62cb62e5 to localhost:59094 2018-01-13 07:25:47,951 DEBUG [RS_CLOSE_REGION-asf911:59071-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.: disabling compactions & flushes 2018-01-13 07:25:47,951 DEBUG [RS:0;asf911:59071] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:47,951 INFO [RS:0;asf911:59071] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:25:47,951 INFO [RS:0;asf911:59071] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:25:47,951 INFO [RS:0;asf911:59071] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:25:47,951 DEBUG [RS_CLOSE_REGION-asf911:59071-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:47,952 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:25:47,952 DEBUG [RS_CLOSE_META-asf911:59071-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:25:47,952 DEBUG [RS:0;asf911:59071] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740, 408ed851e839a1b5b1051712e18f98f9=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.} 2018-01-13 07:25:47,953 DEBUG [RS_CLOSE_META-asf911:59071-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:25:47,953 INFO [StoreCloserThread-hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:47,953 DEBUG [RS_CLOSE_META-asf911:59071-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:25:47,953 ERROR [RS_CLOSE_REGION-asf911:59071-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:25:47,953 INFO [RS_CLOSE_REGION-asf911:59071-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:47,954 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:25:47,954 DEBUG [RS_CLOSE_REGION-asf911:59071-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:47,954 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:25:47,954 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:25:47,954 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:25:47,954 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:25:47,954 ERROR [RS_CLOSE_META-asf911:59071-0] regionserver.HRegion(1620): Memstore size is 1330 2018-01-13 07:25:47,955 DEBUG [RS_CLOSE_META-asf911:59071-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:47,955 INFO [RS_CLOSE_META-asf911:59071-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:47,955 DEBUG [RS_CLOSE_META-asf911:59071-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:25:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:48,153 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,59071,1515828343205; all regions closed. 2018-01-13 07:25:48,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741862_1038{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:48,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741862_1038{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:48,169 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(846): lease for file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta is expired, give up org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta (inode 16563) owned by DFSClient_NONMAPREDUCE_403730989_1 but is accessed by DFSClient_NONMAPREDUCE_-1188838479_1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3430) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3479) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3456) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:787) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:538) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:462) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.complete(Unknown Source) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.completeFile(FanOutOneBlockAsyncDFSOutputHelper.java:837) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:569) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:144) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$executeClose$8(AsyncFSWAL.java:742) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:25:48,175 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:48,175 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:25:48,176 DEBUG [RS:0;asf911:59071] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:25:48,176 INFO [RS:0;asf911:59071] regionserver.Leases(147): RS:0;asf911:59071 closing leases 2018-01-13 07:25:48,176 INFO [RS:0;asf911:59071] regionserver.Leases(150): RS:0;asf911:59071 closed leases 2018-01-13 07:25:48,177 INFO [RS:0;asf911:59071] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,59071,1515828343205 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,59071,1515828343205 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:25:48,177 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:25:48,177 INFO [RS:0;asf911:59071] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:25:48,251 DEBUG [RegionOpenAndInitThread-testRegionServerSessionExpired_1515828347510-1] regionserver.HRegion(807): Instantiated testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:48,251 DEBUG [RegionOpenAndInitThread-testRegionServerSessionExpired_1515828347510-1] regionserver.HRegion(1481): Closing testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.: disabling compactions & flushes 2018-01-13 07:25:48,251 DEBUG [RegionOpenAndInitThread-testRegionServerSessionExpired_1515828347510-1] regionserver.HRegion(1520): Updates disabled for region testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:48,251 INFO [RegionOpenAndInitThread-testRegionServerSessionExpired_1515828347510-1] regionserver.HRegion(1633): Closed testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:48,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:48,406 DEBUG [ProcExecWrkr-15] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:48,407 DEBUG [ProcExecWrkr-15] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.","families":{"info":[{"qualifier":"regioninfo","vlen":78,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:25:48,408 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:48,411 DEBUG [RS-EventLoopGroup-19-19] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59071 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:48,445 INFO [asf911.gq1.ygridcore.net,42210,1515828343058_splitLogManager__Chore_1] master.SplitLogManager$TimeoutMonitor(585): total=1, unassigned=0, tasks={/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta=last_update = 1515828347899 last_version = 2 cur_worker_name = asf911.gq1.ygridcore.net,58799,1515828343253 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2018-01-13 07:25:48,517 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59071 this server is in the failed servers list 2018-01-13 07:25:48,720 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59071 this server is in the failed servers list 2018-01-13 07:25:48,815 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:25:48,815 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:25:48,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:49,023 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59071 this server is in the failed servers list 2018-01-13 07:25:49,417 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-01-13 07:25:49,417 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,59071,1515828343205: regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase regionserver:59071-0x160ee689450002f received expired from ZooKeeper, aborting ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:25:49,418 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:49,422 INFO [main-EventThread] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 519013840 }, "NonHeapMemoryUsage" : { "committed" : 120324096, "init" : 2555904, "max" : -1, "used" : 118614224 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 17222, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 87, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 7, "TotalCallTime_99th_percentile" : 7, "TotalCallTime_99.9th_percentile" : 7, "TotalCallTime_TimeRangeCount_0-1" : 17, "TotalCallTime_TimeRangeCount_1-3" : 9, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 58, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 87, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 55, "ResponseSize_90th_percentile" : 114, "ResponseSize_95th_percentile" : 768, "ResponseSize_98th_percentile" : 864, "ResponseSize_99th_percentile" : 868, "ResponseSize_99.9th_percentile" : 872, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_100000000-inf" : 58, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 87, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 4, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 18, "ProcessCallTime_TimeRangeCount_1-3" : 8, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 14, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 87, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 86, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 130, "RequestSize_90th_percentile" : 377, "RequestSize_95th_percentile" : 517, "RequestSize_98th_percentile" : 734, "RequestSize_99th_percentile" : 735, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 17, "RequestSize_SizeRangeCount_100-1000" : 12, "RequestSize_SizeRangeCount_100000000-inf" : 58, "sentBytes" : 7925, "QueueCallTime_num_ops" : 87, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "QueueCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:49,423 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-01-13 07:25:49,423 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,59071,1515828343205: regionserver:59071-0x160ee689450002f, quorum=localhost:59094, baseZNode=/hbase regionserver:59071-0x160ee689450002f received expired from ZooKeeper, aborting ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:25:49,423 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:25:49,425 INFO [main-EventThread] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 520030168 }, "NonHeapMemoryUsage" : { "committed" : 120324096, "init" : 2555904, "max" : -1, "used" : 118640232 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 17222, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 87, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 7, "TotalCallTime_99th_percentile" : 7, "TotalCallTime_99.9th_percentile" : 7, "TotalCallTime_TimeRangeCount_0-1" : 17, "TotalCallTime_TimeRangeCount_1-3" : 9, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 58, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 87, "ResponseSize_min" : 0, "ResponseSize_max" : 873, "ResponseSize_mean" : 81, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 55, "ResponseSize_90th_percentile" : 114, "ResponseSize_95th_percentile" : 768, "ResponseSize_98th_percentile" : 864, "ResponseSize_99th_percentile" : 868, "ResponseSize_99.9th_percentile" : 872, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 10, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_100000000-inf" : 58, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 87, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 4, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 18, "ProcessCallTime_TimeRangeCount_1-3" : 8, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 14, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 87, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 145, "RequestSize_25th_percentile" : 86, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 130, "RequestSize_90th_percentile" : 377, "RequestSize_95th_percentile" : 517, "RequestSize_98th_percentile" : 734, "RequestSize_99th_percentile" : 735, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 17, "RequestSize_SizeRangeCount_100-1000" : 12, "RequestSize_SizeRangeCount_100000000-inf" : 58, "sentBytes" : 7925, "QueueCallTime_num_ops" : 87, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 29, "QueueCallTime_TimeRangeCount_600000-inf" : 58, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:25:49,518 WARN [RS:0;asf911:59071] regionserver.HRegionServer(1146): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf911.gq1.ygridcore.net,59071,1515828343205 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1591) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1143) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:25:49,518 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,59071,1515828343205; zookeeper connection closed. 2018-01-13 07:25:49,518 INFO [RS:0;asf911:59071] regionserver.HRegionServer(1157): RS:0;asf911:59071 exiting 2018-01-13 07:25:49,519 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7b0442c1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7b0442c1 2018-01-13 07:25:49,525 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59071 this server is in the failed servers list 2018-01-13 07:25:49,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:50,534 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:50,535 DEBUG [RS-EventLoopGroup-19-20] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59071 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:50,537 INFO [hconnection-0x2ec470b9-shared-pool32-t9] client.AsyncRequestFutureImpl(778): id=23, table=hbase:meta, attempt=6/46 failed=1ops, last exception=java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:59071 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 on asf911.gq1.ygridcore.net,59071,1515828343205, tracking started null, retrying after=2007ms, replay=1ops 2018-01-13 07:25:50,592 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:25:50,648 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(156): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:25:51,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:51,912 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] util.FSHDFSUtils(284): Recovered lease, attempt=1 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta after 4013ms 2018-01-13 07:25:51,948 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter(284): DLS Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 5 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "rep_meta" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "rep_position" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2018-01-13 07:25:51,950 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter(336): Finishing writing output logs and closing down 2018-01-13 07:25:51,950 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-01-13 07:25:51,961 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1502): Creating writer path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta.temp 2018-01-13 07:25:51,970 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-01-13 07:25:51,982 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741867_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:51,982 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741867_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:51,984 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1379): Closed wap hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta.temp (wrote 6 edits, skipped 0 edits in 20ms 2018-01-13 07:25:51,987 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1405): Rename hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta.temp to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000010 2018-01-13 07:25:51,987 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] wal.WALSplitter(357): Processed 6 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta, length=0, corrupted=false, progress failed=false 2018-01-13 07:25:52,005 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,005 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,005 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta to final state DONE asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:52,006 INFO [RS_LOG_REPLAY_OPS-asf911:58799-0] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,58799,1515828343253 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@56be11ed in 4146ms. Status = DONE 2018-01-13 07:25:52,006 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(371): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta entered state: DONE asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:52,018 INFO [main-EventThread] wal.WALSplitter(449): Archived processed log hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,019 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(414): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,027 WARN [ProcExecWrkr-14] master.SplitLogManager(273): Returning success without actually splitting and deleting all the log files in path hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting: [FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869; isDirectory=false; length=599; replication=2; blocksize=134217728; modification_time=1515828348175; access_time=1515828344873; owner=jenkins.hfs.6; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting is non empty': Directory is not empty at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:84) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3676) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:623) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:267) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:293) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:232) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:224) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:110) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:51) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1456) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1225) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1735) 2018-01-13 07:25:52,028 INFO [ProcExecWrkr-14] master.SplitLogManager(287): finished splitting (more than or equal to) 0 bytes in 1 log files in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting] in 4207ms 2018-01-13 07:25:52,030 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,030 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:52,031 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(694): deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.meta.1515828345071.meta 2018-01-13 07:25:52,230 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=9, ppid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true 2018-01-13 07:25:52,230 INFO [ProcExecWrkr-14] procedure.RecoverMetaProcedure(130): pid=9, ppid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true; Assigning meta with new plan. previous meta server=asf911.gq1.ygridcore.net,59071,1515828343205 2018-01-13 07:25:52,231 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:25:52,406 INFO [ProcExecWrkr-14] procedure.MasterProcedureScheduler(883): pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:25:52,406 INFO [ProcExecWrkr-14] assignment.AssignProcedure(211): Start pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:25:52,546 DEBUG [hconnection-0x2ec470b9-shared-pool32-t9] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:52,548 DEBUG [RS-EventLoopGroup-19-21] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59071 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:52,550 INFO [hconnection-0x2ec470b9-shared-pool32-t9] client.AsyncRequestFutureImpl(778): id=23, table=hbase:meta, attempt=7/46 failed=1ops, last exception=java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:59071 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 on asf911.gq1.ygridcore.net,59071,1515828343205, tracking started null, retrying after=4009ms, replay=1ops 2018-01-13 07:25:52,556 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=1, allServersCount=1 2018-01-13 07:25:52,557 INFO [ProcExecWrkr-3] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:52,597 INFO [ProcExecWrkr-3] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:52,748 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,58799,1515828343253 version=3145728 2018-01-13 07:25:52,749 DEBUG [RSProcedureDispatcher-pool29-t3] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:52,750 DEBUG [RSProcedureDispatcher-pool29-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:25:52,752 INFO [RS-EventLoopGroup-19-23] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:52,752 INFO [RS-EventLoopGroup-19-23] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56699 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:52,754 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58799] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:25:52,755 INFO [RS_OPEN_META-asf911:58799-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:25:52,759 INFO [RS_OPEN_META-asf911:58799-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C58799%2C1515828343253.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:25:52,765 DEBUG [RS-EventLoopGroup-19-24] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:25:52,765 DEBUG [RS-EventLoopGroup-19-25] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:25:52,768 INFO [RS_OPEN_META-asf911:58799-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253/asf911.gq1.ygridcore.net%2C58799%2C1515828343253.meta.1515828352759.meta 2018-01-13 07:25:52,768 DEBUG [RS_OPEN_META-asf911:58799-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:25:52,769 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:52,769 DEBUG [RS_OPEN_META-asf911:58799-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:25:52,769 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:25:52,770 INFO [RS_OPEN_META-asf911:58799-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:25:52,770 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:25:52,770 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:25:52,774 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:52,774 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:25:52,774 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:52,775 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:52,775 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:52,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:52,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:25:52,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:52,778 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:52,779 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:52,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:52,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:25:52,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:52,782 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:52,783 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:52,785 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:52,786 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:25:52,786 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:52,786 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:52,786 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:52,789 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:52,789 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:25:52,790 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:52,790 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:52,790 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:52,795 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(4350): Found 1 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:25:52,797 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(4447): Replaying edits from hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000010 2018-01-13 07:25:52,811 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(4634): Applied 13, skipped 0, firstSequenceIdInLog=5, maxSequenceIdInLog=10, path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000010 2018-01-13 07:25:52,812 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(2516): Flushing 5/5 column families, memstore=1.30 KB 2018-01-13 07:25:52,876 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741869_1046{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 5961 2018-01-13 07:25:52,877 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741869_1046 size 5961 2018-01-13 07:25:53,278 INFO [RS_OPEN_META-asf911:58799-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=10, memsize=1.2 K, hasBloomFilter=false, into tmp file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/info/14369337a4f744a5a1f35c7bc16349eb 2018-01-13 07:25:53,369 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741870_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:53,370 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741870_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:53,371 INFO [RS_OPEN_META-asf911:58799-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=10, memsize=136, hasBloomFilter=false, into tmp file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/table/92b0443de86e478688287256be22296b 2018-01-13 07:25:53,383 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/info/14369337a4f744a5a1f35c7bc16349eb as hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info/14369337a4f744a5a1f35c7bc16349eb 2018-01-13 07:25:53,395 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HStore(1028): Added hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info/14369337a4f744a5a1f35c7bc16349eb, entries=10, sequenceid=10, filesize=5.8 K 2018-01-13 07:25:53,405 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/table/92b0443de86e478688287256be22296b as hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table/92b0443de86e478688287256be22296b 2018-01-13 07:25:53,416 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HStore(1028): Added hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table/92b0443de86e478688287256be22296b, entries=3, sequenceid=10, filesize=4.7 K 2018-01-13 07:25:53,416 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(2712): Finished memstore flush of ~1.30 KB/1330, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 604ms, sequenceid=10, compaction requested=false; wal=null 2018-01-13 07:25:53,418 INFO [IPC Server handler 9 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741867_1044 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:53,419 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(4427): Deleted recovered.edits file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000010 2018-01-13 07:25:53,421 DEBUG [RS_OPEN_META-asf911:58799-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:25:53,425 DEBUG [RS_OPEN_META-asf911:58799-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/11.seqid, newSeqId=11, maxSeqId=3 2018-01-13 07:25:53,426 INFO [RS_OPEN_META-asf911:58799-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=11 2018-01-13 07:25:53,436 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:25:53,437 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=42210] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=11, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:53,438 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828353437}]},"ts":1515828353437} 2018-01-13 07:25:53,438 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:25:53,440 DEBUG [RS_OPEN_META-asf911:58799-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:53,442 DEBUG [hconnection-0x2ec470b9-shared-pool32-t10] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:25:53,443 INFO [RS-EventLoopGroup-19-29] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:53,444 INFO [RS-EventLoopGroup-19-29] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56708 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:53,447 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:25:53,447 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:53,623 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=9, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:25:53,778 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=9, ppid=7, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true; resume parent processing. 2018-01-13 07:25:53,779 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=9, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.2340sec 2018-01-13 07:25:53,931 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=9, ppid=7, state=SUCCESS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true 2018-01-13 07:25:54,087 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true; resume parent processing. 2018-01-13 07:25:54,087 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:54,087 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=9, ppid=7, state=SUCCESS; RecoverMetaProcedure failedMetaServer=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true in 6.0200sec 2018-01-13 07:25:54,243 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:54,244 DEBUG [ProcExecWrkr-6] procedure.ServerCrashProcedure(218): Splitting WALs pid=7, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:54,246 INFO [ProcExecWrkr-6] master.SplitLogManager(455): dead splitlog workers [asf911.gq1.ygridcore.net,59071,1515828343205] 2018-01-13 07:25:54,249 INFO [ProcExecWrkr-6] master.SplitLogManager(235): Started splitting 1 logs in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting] for [asf911.gq1.ygridcore.net,59071,1515828343205] 2018-01-13 07:25:54,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:54,265 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(331): put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,266 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(363): task not yet acquired /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 ver = 0 2018-01-13 07:25:54,266 DEBUG [SplitLogWorker-asf911:58799] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,58799,1515828343253 is ready to take more tasks, will get task list and try grab tasks again. 2018-01-13 07:25:54,273 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,273 INFO [SplitLogWorker-asf911:58799] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,58799,1515828343253 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,274 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(453): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 acquired by asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:54,300 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter(251): Splitting WAL=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869, length=599 2018-01-13 07:25:54,307 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,307 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869 2018-01-13 07:25:54,307 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,308 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869 after 1ms 2018-01-13 07:25:54,318 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter(284): DLS Last flushed sequenceid for 408ed851e839a1b5b1051712e18f98f9: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2018-01-13 07:25:54,319 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter(336): Finishing writing output logs and closing down 2018-01-13 07:25:54,319 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-01-13 07:25:54,325 DEBUG [RS_LOG_REPLAY_OPS-asf911:58799-1-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1502): Creating writer path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869.temp 2018-01-13 07:25:54,326 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-01-13 07:25:54,337 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741871_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:25:54,337 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741871_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:25:54,338 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1379): Closed wap hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869.temp (wrote 2 edits, skipped 0 edits in 7ms 2018-01-13 07:25:54,341 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1405): Rename hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869.temp to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000005 2018-01-13 07:25:54,341 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] wal.WALSplitter(357): Processed 2 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869, length=599, corrupted=false, progress failed=false 2018-01-13 07:25:54,348 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,348 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,348 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 to final state DONE asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:54,349 INFO [RS_LOG_REPLAY_OPS-asf911:58799-1] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,58799,1515828343253 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@85b3eaf in 75ms. Status = DONE 2018-01-13 07:25:54,349 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(371): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 entered state: DONE asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:54,354 INFO [main-EventThread] wal.WALSplitter(449): Archived processed log hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869 to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs/asf911.gq1.ygridcore.net%2C59071%2C1515828343205.1515828344869 2018-01-13 07:25:54,355 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(414): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,356 INFO [ProcExecWrkr-6] master.SplitLogManager(287): finished splitting (more than or equal to) 599 bytes in 1 log files in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59071,1515828343205-splitting] in 107ms 2018-01-13 07:25:54,358 DEBUG [ProcExecWrkr-6] procedure.ServerCrashProcedure(226): Done splitting WALs pid=7, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:54,365 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,365 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,365 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:25:54,365 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(694): deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59071%2C1515828343205-splitting%2Fasf911.gq1.ygridcore.net%252C59071%252C1515828343205.1515828344869 2018-01-13 07:25:54,514 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:54,517 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9}] 2018-01-13 07:25:54,670 INFO [ProcExecWrkr-6] procedure.MasterProcedureScheduler(883): pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9 hbase:namespace hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:54,691 INFO [ProcExecWrkr-6] assignment.AssignProcedure(211): Start pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59071,1515828343205; forceNewPlan=false, retain=true 2018-01-13 07:25:54,842 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:54,844 DEBUG [MultiHConnection-shared--pool33-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:54,845 DEBUG [RS-EventLoopGroup-19-30] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59071 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:25:54,951 DEBUG [MultiHConnection-shared--pool33-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:25:54,953 INFO [RS-EventLoopGroup-19-32] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:25:54,953 INFO [RS-EventLoopGroup-19-32] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56716 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:25:54,958 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(208): Dispatch pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:55,108 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,58799,1515828343253 version=3145728 2018-01-13 07:25:55,112 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58799] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:55,118 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(6949): Opening region: {ENCODED => 408ed851e839a1b5b1051712e18f98f9, NAME => 'hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:55,118 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 408ed851e839a1b5b1051712e18f98f9 2018-01-13 07:25:55,118 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:55,123 DEBUG [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info 2018-01-13 07:25:55,123 DEBUG [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info 2018-01-13 07:25:55,124 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:25:55,124 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=1, currentSize=766408, freeSize=1043195896, maxSize=1043962304, heapSize=766408, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:55,124 INFO [StoreOpener-408ed851e839a1b5b1051712e18f98f9-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:55,129 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(4350): Found 1 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9 2018-01-13 07:25:55,131 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(4447): Replaying edits from hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000005 2018-01-13 07:25:55,137 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(4634): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000005 2018-01-13 07:25:55,137 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=78 B 2018-01-13 07:25:55,164 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741872_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:25:55,165 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741872_1049 size 4898 2018-01-13 07:25:55,166 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=5, memsize=78, hasBloomFilter=true, into tmp file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/.tmp/info/3d1ae35732fc4b96b9cb70705ee15991 2018-01-13 07:25:55,181 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/.tmp/info/3d1ae35732fc4b96b9cb70705ee15991 as hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info/3d1ae35732fc4b96b9cb70705ee15991 2018-01-13 07:25:55,194 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HStore(1028): Added hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/info/3d1ae35732fc4b96b9cb70705ee15991, entries=2, sequenceid=5, filesize=4.8 K 2018-01-13 07:25:55,196 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(2712): Finished memstore flush of ~78 B/78, currentsize=0 B/0 for region hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. in 58ms, sequenceid=5, compaction requested=false; wal=null 2018-01-13 07:25:55,196 INFO [IPC Server handler 9 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741871_1048 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:25:55,197 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(4427): Deleted recovered.edits file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/0000000000000000005 2018-01-13 07:25:55,205 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/408ed851e839a1b5b1051712e18f98f9/recovered.edits/6.seqid, newSeqId=6, maxSeqId=2 2018-01-13 07:25:55,206 INFO [RS_OPEN_PRIORITY_REGION-asf911:58799-0] regionserver.HRegion(960): Onlined 408ed851e839a1b5b1051712e18f98f9; next sequenceid=6 2018-01-13 07:25:55,208 INFO [PostOpenDeployTasks:408ed851e839a1b5b1051712e18f98f9] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:55,210 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=42210] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=6, pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:55,210 INFO [ProcExecWrkr-11] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9., regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:55,210 DEBUG [PostOpenDeployTasks:408ed851e839a1b5b1051712e18f98f9] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:25:55,212 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:58799-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. on asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:55,373 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=11, ppid=7, state=SUCCESS; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9 2018-01-13 07:25:55,535 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:SERVER_CRASH_FINISH; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true; resume parent processing. 2018-01-13 07:25:55,535 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:SERVER_CRASH_FINISH; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true 2018-01-13 07:25:55,535 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=11, ppid=7, state=SUCCESS; AssignProcedure table=hbase:namespace, region=408ed851e839a1b5b1051712e18f98f9 in 700msec 2018-01-13 07:25:55,693 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59071,1515828343205, splitWal=true, meta=true in 8.1070sec 2018-01-13 07:25:55,825 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741867_1044, blk_1073741871_1048] 2018-01-13 07:25:55,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:25:56,153 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:25:56,211 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:25:56,212 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:25:56,565 INFO [ProcExecWrkr-15] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:25:56,565 INFO [hconnection-0x2ec470b9-shared-pool32-t9] client.AsyncRequestFutureImpl(805): id=23, table=hbase:meta, attempt=8/46 succeeded on asf911.gq1.ygridcore.net,58799,1515828343253, tracking started Sat Jan 13 07:25:48 UTC 2018 2018-01-13 07:25:56,650 DEBUG [ProcExecWrkr-15] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:56,650 DEBUG [ProcExecWrkr-15] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testRegionServerSessionExpired_1515828347510","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828356650}]},"ts":1515828356650} 2018-01-13 07:25:56,656 INFO [ProcExecWrkr-15] hbase.MetaTableAccessor(1786): Updated table testRegionServerSessionExpired_1515828347510 state to ENABLING in META 2018-01-13 07:25:56,656 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=12, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253}] 2018-01-13 07:25:56,707 INFO [ProcExecWrkr-13] procedure.MasterProcedureScheduler(883): pid=12, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253 testRegionServerSessionExpired_1515828347510 testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:56,711 INFO [ProcExecWrkr-13] assignment.AssignProcedure(211): Start pid=12, ppid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58799,1515828343253; forceNewPlan=false, retain=false 2018-01-13 07:25:56,863 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:56,877 INFO [ProcExecWrkr-16] assignment.RegionTransitionProcedure(208): Dispatch pid=12, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:57,027 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,58799,1515828343253 version=3145728 2018-01-13 07:25:57,029 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58799] regionserver.RSRpcServices(1907): Open testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:57,035 DEBUG [RS_OPEN_REGION-asf911:58799-0] regionserver.HRegion(6949): Opening region: {ENCODED => 116dec78576bc92f3d7926bfc5529c56, NAME => 'testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:25:57,036 DEBUG [RS_OPEN_REGION-asf911:58799-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionServerSessionExpired_1515828347510 116dec78576bc92f3d7926bfc5529c56 2018-01-13 07:25:57,036 DEBUG [RS_OPEN_REGION-asf911:58799-0] regionserver.HRegion(807): Instantiated testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:57,043 DEBUG [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionServerSessionExpired_1515828347510/116dec78576bc92f3d7926bfc5529c56/fam 2018-01-13 07:25:57,044 DEBUG [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionServerSessionExpired_1515828347510/116dec78576bc92f3d7926bfc5529c56/fam 2018-01-13 07:25:57,061 INFO [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:25:57,061 INFO [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:25:57,061 INFO [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] hfile.CacheConfig(239): Created cacheConfig for fam: blockCache=LruBlockCache{blockCount=1, currentSize=766408, freeSize=1043195896, maxSize=1043962304, heapSize=766408, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:25:57,062 INFO [StoreOpener-116dec78576bc92f3d7926bfc5529c56-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:25:57,064 INFO [RS_OPEN_REGION-asf911:58799-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:57,065 DEBUG [RS_OPEN_REGION-asf911:58799-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionServerSessionExpired_1515828347510/116dec78576bc92f3d7926bfc5529c56 2018-01-13 07:25:57,068 DEBUG [RS_OPEN_REGION-asf911:58799-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionServerSessionExpired_1515828347510 descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:25:57,073 DEBUG [RS_OPEN_REGION-asf911:58799-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionServerSessionExpired_1515828347510/116dec78576bc92f3d7926bfc5529c56/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:25:57,073 INFO [RS_OPEN_REGION-asf911:58799-0] regionserver.HRegion(960): Onlined 116dec78576bc92f3d7926bfc5529c56; next sequenceid=2 2018-01-13 07:25:57,075 INFO [PostOpenDeployTasks:116dec78576bc92f3d7926bfc5529c56] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:57,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=12, ppid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253; rit=OPENING, location=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:57,078 INFO [ProcExecWrkr-2] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:57,078 DEBUG [PostOpenDeployTasks:116dec78576bc92f3d7926bfc5529c56] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:25:57,080 DEBUG [RS_OPEN_REGION-asf911:58799-0] handler.OpenRegionHandler(128): Opened testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. on asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:57,130 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=12, ppid=8, state=SUCCESS; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:25:57,172 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=8, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510; resume parent processing. 2018-01-13 07:25:57,172 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:57,172 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=12, ppid=8, state=SUCCESS; AssignProcedure table=testRegionServerSessionExpired_1515828347510, region=116dec78576bc92f3d7926bfc5529c56, target=asf911.gq1.ygridcore.net,58799,1515828343253 in 428msec 2018-01-13 07:25:57,173 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testRegionServerSessionExpired_1515828347510","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828357172}]},"ts":1515828357172} 2018-01-13 07:25:57,183 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1786): Updated table testRegionServerSessionExpired_1515828347510 state to ENABLED in META 2018-01-13 07:25:57,254 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=8, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 2018-01-13 07:25:57,301 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=8, state=SUCCESS; CreateTableProcedure table=testRegionServerSessionExpired_1515828347510 in 9.7060sec 2018-01-13 07:25:58,826 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741867_1044, blk_1073741871_1048] 2018-01-13 07:26:00,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:00,434 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-13 07:26:02,541 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:02,584 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:05,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(1157): Checking to see if procedure is done pid=8 2018-01-13 07:26:05,901 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3638): Operation: CREATE, Table Name: default:testRegionServerSessionExpired_1515828347510 completed 2018-01-13 07:26:05,901 INFO [Time-limited test] hbase.TestZooKeeper(176): Putting table testRegionServerSessionExpired_1515828347510 2018-01-13 07:26:05,904 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:26:05,906 DEBUG [Default-IPC-NioEventLoopGroup-7-9] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59071 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59071 2018-01-13 07:26:06,011 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:26:06,013 INFO [RS-EventLoopGroup-19-1] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:06,014 INFO [RS-EventLoopGroup-19-1] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56768 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:06,028 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,59071,1515828343205: killAll ***** 2018-01-13 07:26:06,029 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:06,035 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 614656128 }, "NonHeapMemoryUsage" : { "committed" : 125239296, "init" : 2555904, "max" : -1, "used" : 123305296 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 28216, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 130, "TotalCallTime_min" : 1, "TotalCallTime_max" : 6, "TotalCallTime_mean" : 2, "TotalCallTime_25th_percentile" : 2, "TotalCallTime_median" : 3, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 6, "TotalCallTime_95th_percentile" : 6, "TotalCallTime_98th_percentile" : 6, "TotalCallTime_99th_percentile" : 6, "TotalCallTime_99.9th_percentile" : 6, "TotalCallTime_TimeRangeCount_0-1" : 1, "TotalCallTime_TimeRangeCount_1-3" : 5, "TotalCallTime_TimeRangeCount_3-10" : 1, "TotalCallTime_TimeRangeCount_600000-inf" : 123, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 130, "ResponseSize_min" : 0, "ResponseSize_max" : 86, "ResponseSize_mean" : 19, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 10, "ResponseSize_90th_percentile" : 86, "ResponseSize_95th_percentile" : 86, "ResponseSize_98th_percentile" : 86, "ResponseSize_99th_percentile" : 86, "ResponseSize_99.9th_percentile" : 86, "ResponseSize_SizeRangeCount_0-10" : 6, "ResponseSize_SizeRangeCount_10-100" : 1, "ResponseSize_SizeRangeCount_100000000-inf" : 123, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 130, "ProcessCallTime_min" : 1, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 2, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 6, "ProcessCallTime_95th_percentile" : 6, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 1, "ProcessCallTime_TimeRangeCount_1-3" : 5, "ProcessCallTime_TimeRangeCount_3-10" : 1, "ProcessCallTime_TimeRangeCount_600000-inf" : 123, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 21, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 130, "RequestSize_min" : 116, "RequestSize_max" : 950, "RequestSize_mean" : 393, "RequestSize_25th_percentile" : 125, "RequestSize_median" : 191, "RequestSize_75th_percentile" : 671, "RequestSize_90th_percentile" : 805, "RequestSize_95th_percentile" : 877, "RequestSize_98th_percentile" : 921, "RequestSize_99th_percentile" : 935, "RequestSize_99.9th_percentile" : 948, "RequestSize_SizeRangeCount_100-1000" : 7, "RequestSize_SizeRangeCount_100000000-inf" : 123, "sentBytes" : 10914, "QueueCallTime_num_ops" : 130, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 7, "QueueCallTime_TimeRangeCount_600000-inf" : 123, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043195896, "blockCacheCount" : 1, "blockCacheSize" : 766408, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 1, "blockCacheMissCountPrimary" : 1, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 1, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 1, "l1CacheHitRatio" : 0.0, "l1CacheMissRatio" : 1.0, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:06,036 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,58799,1515828343253: killAll ***** 2018-01-13 07:26:06,036 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:06,038 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1208483840, "init" : 792723456, "max" : 2609905664, "used" : 614656128 }, "NonHeapMemoryUsage" : { "committed" : 125239296, "init" : 2555904, "max" : -1, "used" : 123305296 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 28216, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 130, "TotalCallTime_min" : 1, "TotalCallTime_max" : 6, "TotalCallTime_mean" : 2, "TotalCallTime_25th_percentile" : 2, "TotalCallTime_median" : 3, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 6, "TotalCallTime_95th_percentile" : 6, "TotalCallTime_98th_percentile" : 6, "TotalCallTime_99th_percentile" : 6, "TotalCallTime_99.9th_percentile" : 6, "TotalCallTime_TimeRangeCount_0-1" : 1, "TotalCallTime_TimeRangeCount_1-3" : 5, "TotalCallTime_TimeRangeCount_3-10" : 1, "TotalCallTime_TimeRangeCount_600000-inf" : 123, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 130, "ResponseSize_min" : 0, "ResponseSize_max" : 86, "ResponseSize_mean" : 19, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 10, "ResponseSize_90th_percentile" : 86, "ResponseSize_95th_percentile" : 86, "ResponseSize_98th_percentile" : 86, "ResponseSize_99th_percentile" : 86, "ResponseSize_99.9th_percentile" : 86, "ResponseSize_SizeRangeCount_0-10" : 6, "ResponseSize_SizeRangeCount_10-100" : 1, "ResponseSize_SizeRangeCount_100000000-inf" : 123, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 130, "ProcessCallTime_min" : 1, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 2, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 6, "ProcessCallTime_95th_percentile" : 6, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 1, "ProcessCallTime_TimeRangeCount_1-3" : 5, "ProcessCallTime_TimeRangeCount_3-10" : 1, "ProcessCallTime_TimeRangeCount_600000-inf" : 123, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 21, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 130, "RequestSize_min" : 116, "RequestSize_max" : 950, "RequestSize_mean" : 393, "RequestSize_25th_percentile" : 125, "RequestSize_median" : 191, "RequestSize_75th_percentile" : 671, "RequestSize_90th_percentile" : 805, "RequestSize_95th_percentile" : 877, "RequestSize_98th_percentile" : 921, "RequestSize_99th_percentile" : 935, "RequestSize_99.9th_percentile" : 948, "RequestSize_SizeRangeCount_100-1000" : 7, "RequestSize_SizeRangeCount_100000000-inf" : 123, "sentBytes" : 10914, "QueueCallTime_num_ops" : 130, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 7, "QueueCallTime_TimeRangeCount_600000-inf" : 123, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043195896, "blockCacheCount" : 1, "blockCacheSize" : 766408, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 1, "blockCacheMissCountPrimary" : 1, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 1, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 1, "l1CacheHitRatio" : 0.0, "l1CacheMissRatio" : 1.0, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:06,041 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42210] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,58799,1515828343253 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,58799,1515828343253: killAll ***** 2018-01-13 07:26:06,042 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,58799,1515828343253' ***** 2018-01-13 07:26:06,042 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:06,042 INFO [RS:1;asf911:58799] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:06,042 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:06,043 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:06,043 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,42210,1515828343058' ***** 2018-01-13 07:26:06,043 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:06,042 INFO [SplitLogWorker-asf911:58799] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:06,042 INFO [RS:1;asf911:58799] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:06,045 INFO [SplitLogWorker-asf911:58799] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,58799,1515828343253 exiting 2018-01-13 07:26:06,045 DEBUG [M:0;asf911:42210] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1db29477, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:06,045 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:06,045 INFO [RS:1;asf911:58799] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:06,047 INFO [M:0;asf911:42210] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:26:06,047 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:06,048 DEBUG [M:0;asf911:42210] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:06,048 INFO [RS:1;asf911:58799] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:06,048 INFO [M:0;asf911:42210] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x0b6462ae to localhost:59094 2018-01-13 07:26:06,048 DEBUG [M:0;asf911:42210] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:06,048 INFO [M:0;asf911:42210] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,42210,1515828343058; all regions closed. 2018-01-13 07:26:06,049 DEBUG [M:0;asf911:42210] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:06,049 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:26:06,049 DEBUG [RS_CLOSE_REGION-asf911:58799-1] handler.CloseRegionHandler(92): Processing close of testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:26:06,048 DEBUG [RS_CLOSE_REGION-asf911:58799-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:26:06,049 INFO [M:0;asf911:42210] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,42210,1515828343058 had [[ScheduledChore: Name: CatalogJanitor-asf911:42210 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42210,1515828343058-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42210,1515828343058-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42210,1515828343058-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42210,1515828343058-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42210,1515828343058-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:06,049 DEBUG [RS:1;asf911:58799] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:06,051 DEBUG [RS_CLOSE_REGION-asf911:58799-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.: disabling compactions & flushes 2018-01-13 07:26:06,050 DEBUG [RS_CLOSE_REGION-asf911:58799-1] regionserver.HRegion(1481): Closing testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.: disabling compactions & flushes 2018-01-13 07:26:06,052 DEBUG [RS_CLOSE_REGION-asf911:58799-1] regionserver.HRegion(1520): Updates disabled for region testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:26:06,051 DEBUG [RS_CLOSE_REGION-asf911:58799-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:26:06,051 INFO [RS:1;asf911:58799] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x70b71f65 to localhost:59094 2018-01-13 07:26:06,051 INFO [M:0;asf911:42210] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:06,052 INFO [M:0;asf911:42210] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:06,052 DEBUG [M:0;asf911:42210] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:06,052 INFO [StoreCloserThread-testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.-1] regionserver.HStore(930): Closed fam 2018-01-13 07:26:06,052 DEBUG [RS:1;asf911:58799] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:06,053 INFO [RS:1;asf911:58799] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:06,053 INFO [RS:1;asf911:58799] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:06,053 INFO [RS:1;asf911:58799] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:06,053 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1388): Waiting on 3 regions to close 2018-01-13 07:26:06,053 DEBUG [RS_CLOSE_META-asf911:58799-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:06,053 DEBUG [RS:1;asf911:58799] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740, 408ed851e839a1b5b1051712e18f98f9=hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9., 116dec78576bc92f3d7926bfc5529c56=testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56.} 2018-01-13 07:26:06,055 DEBUG [RS_CLOSE_META-asf911:58799-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:06,055 DEBUG [RS_CLOSE_META-asf911:58799-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:06,053 ERROR [RS_CLOSE_REGION-asf911:58799-1] regionserver.HRegion(1620): Memstore size is 41 2018-01-13 07:26:06,055 INFO [StoreCloserThread-hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9.-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:06,056 INFO [RS_CLOSE_REGION-asf911:58799-1] regionserver.HRegion(1633): Closed testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:26:06,056 DEBUG [RS_CLOSE_REGION-asf911:58799-1] handler.CloseRegionHandler(125): Closed testRegionServerSessionExpired_1515828347510,,1515828347543.116dec78576bc92f3d7926bfc5529c56. 2018-01-13 07:26:06,058 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:06,059 INFO [RS_CLOSE_REGION-asf911:58799-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:26:06,059 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:06,059 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:06,059 DEBUG [RS_CLOSE_REGION-asf911:58799-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828345301.408ed851e839a1b5b1051712e18f98f9. 2018-01-13 07:26:06,059 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:06,061 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:06,061 ERROR [RS_CLOSE_META-asf911:58799-0] regionserver.HRegion(1620): Memstore size is 2745 2018-01-13 07:26:06,061 DEBUG [RS_CLOSE_META-asf911:58799-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:06,063 INFO [RS_CLOSE_META-asf911:58799-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:06,063 DEBUG [RS_CLOSE_META-asf911:58799-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:06,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:06,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:06,065 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:06,066 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,50456,1515828343146' ***** 2018-01-13 07:26:06,066 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:06,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:06,067 INFO [M:1;asf911:50456] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x444862ec to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:06,109 DEBUG [M:0;asf911:42210] zookeeper.ZKUtil(614): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:06,109 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:06,109 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42210-0x160ee689450002d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:06,109 WARN [M:0;asf911:42210] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:06,109 INFO [M:0;asf911:42210] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:06,109 DEBUG [main] zookeeper.ZKUtil(614): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:06,109 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:06,110 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:06,110 INFO [M:0;asf911:42210] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x528738c4 to localhost:59094 2018-01-13 07:26:06,111 DEBUG [M:0;asf911:42210] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:06,111 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:26:06,112 INFO [M:0;asf911:42210] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:06,113 INFO [M:0;asf911:42210] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:06,113 INFO [M:0;asf911:42210] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:06,115 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,115 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,116 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,117 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,117 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,117 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,120 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,120 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,120 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,119 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,119 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,117 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,122 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,122 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,122 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,122 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:06,134 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:06,134 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:06,135 INFO [M:0;asf911:42210] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,42210,1515828343058_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:06,136 INFO [M:0;asf911:42210] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:06,136 INFO [M:0;asf911:42210] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:06,173 DEBUG [M:0;asf911:42210] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,42210,1515828343058 already deleted, retry=false 2018-01-13 07:26:06,173 DEBUG [main] zookeeper.ZKUtil(614): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:06,174 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:06,175 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,42210,1515828343058 2018-01-13 07:26:06,174 DEBUG [M:1;asf911:50456] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a8f9415, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:06,205 INFO [M:0;asf911:42210] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,42210,1515828343058; zookeeper connection closed. 2018-01-13 07:26:06,206 INFO [M:0;asf911:42210] regionserver.HRegionServer(1157): M:0;asf911:42210 exiting 2018-01-13 07:26:06,207 ERROR [main] master.HMaster(2673): ZooKeeper exception trying to set cluster as down in ZK org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.zookeeper.ClusterStatusTracker.setClusterDown(ClusterStatusTracker.java:84) at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:2671) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:256) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:06,255 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,58799,1515828343253; all regions closed. 2018-01-13 07:26:06,262 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741868_1045{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 3563 2018-01-13 07:26:06,262 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253/asf911.gq1.ygridcore.net%2C58799%2C1515828343253.meta.1515828352759.meta not finished, retry = 0 2018-01-13 07:26:06,262 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741868_1045 size 3563 2018-01-13 07:26:06,369 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741860_1036{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 934 2018-01-13 07:26:06,369 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,58799,1515828343253/asf911.gq1.ygridcore.net%2C58799%2C1515828343253.1515828344868 not finished, retry = 0 2018-01-13 07:26:06,369 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741860_1036 size 934 2018-01-13 07:26:06,471 DEBUG [RS:1;asf911:58799] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:06,471 INFO [RS:1;asf911:58799] regionserver.Leases(147): RS:1;asf911:58799 closing leases 2018-01-13 07:26:06,471 INFO [RS:1;asf911:58799] regionserver.Leases(150): RS:1;asf911:58799 closed leases 2018-01-13 07:26:06,471 INFO [RS:1;asf911:58799] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,58799,1515828343253 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,58799,1515828343253 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:06,472 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:06,472 INFO [RS:1;asf911:58799] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:06,482 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,58799,1515828343253 2018-01-13 07:26:06,482 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:58799-0x160ee6894500030, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:06,490 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,58799,1515828343253; zookeeper connection closed. 2018-01-13 07:26:06,490 INFO [RS:1;asf911:58799] regionserver.HRegionServer(1157): RS:1;asf911:58799 exiting 2018-01-13 07:26:06,491 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@67eb9768] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@67eb9768 2018-01-13 07:26:06,491 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:26:06,817 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:06,818 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:07,178 INFO [M:1;asf911:50456] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:07,179 INFO [M:1;asf911:50456] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,50456,1515828343146 2018-01-13 07:26:07,181 DEBUG [M:1;asf911:50456] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:07,181 INFO [M:1;asf911:50456] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x444862ec to localhost:59094 2018-01-13 07:26:07,181 DEBUG [M:1;asf911:50456] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:07,181 INFO [M:1;asf911:50456] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,50456,1515828343146; all regions closed. 2018-01-13 07:26:07,182 INFO [M:1;asf911:50456] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,50456,1515828343146 had [] on shutdown 2018-01-13 07:26:07,182 DEBUG [M:1;asf911:50456] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:07,191 DEBUG [M:1;asf911:50456] zookeeper.ZKUtil(614): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:07,191 WARN [M:1;asf911:50456] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:07,198 DEBUG [M:1;asf911:50456] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,50456,1515828343146 already deleted, retry=false 2018-01-13 07:26:07,207 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50456-0x160ee689450002e, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,50456,1515828343146 2018-01-13 07:26:07,207 INFO [M:1;asf911:50456] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,50456,1515828343146; zookeeper connection closed. 2018-01-13 07:26:07,208 INFO [M:1;asf911:50456] regionserver.HRegionServer(1157): M:1;asf911:50456 exiting 2018-01-13 07:26:07,216 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,219 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741859_1035 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,219 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741860_1036 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,219 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741868_1045 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741865_1041 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741866_1042 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741857_1033 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741869_1046 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,220 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741870_1047 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,221 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741863_1039 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,221 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741864_1040 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,221 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741872_1049 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,221 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741856_1032 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,222 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741855_1031 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,222 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741861_1037 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:07,222 INFO [IPC Server handler 7 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741862_1043 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:07,223 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:07,232 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:07,233 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500039 connected 2018-01-13 07:26:07,234 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:26:07,234 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:26:07,235 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:26:07,235 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:26:07,236 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:26:07,236 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:26:07,237 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:26:07,237 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:26:07,237 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:26:07,238 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-01-13 07:26:07,239 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:26:07,239 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:07,240 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:26:07,240 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:26:07,243 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:26:07,244 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:26:07,244 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:26:07,245 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:26:07,246 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:26:07,246 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:26:07,248 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:26:07,248 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:26:07,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:07,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:07,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:07,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:26:07,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:26:07,258 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:07,259 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:26:07,264 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:07,264 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:26:07,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:26:07,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:07,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:07,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:07,265 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500039, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:26:07,296 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testRegionServerSessionExpired Thread=347 (was 281) Potentially hanging thread: RS-EventLoopGroup-19-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-24 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-31 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-18-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42210-HFileCleaner.large.0-1515828343632 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-19-32 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-18-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42210-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-17-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-18-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:641) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.6@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-30 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.7@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.7 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-19-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-29 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-18-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-25 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42210-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-19-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-27 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42210-HFileCleaner.small.0-1515828343632 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-18-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-23 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-28 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-16-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-788220821_1 at /127.0.0.1:40560 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-26 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-19-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2445 (was 2055) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 199) - SystemLoadAverage LEAK? -, ProcessCount=270 (was 270), AvailableMemoryMB=11469 (was 10829) - AvailableMemoryMB LEAK? - 2018-01-13 07:26:07,297 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2445 is superior to 1024 2018-01-13 07:26:07,325 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testMasterSessionExpired Thread=346, OpenFileDescriptor=2437, MaxFileDescriptor=60000, SystemLoadAverage=200, ProcessCount=270, AvailableMemoryMB=11461 2018-01-13 07:26:07,325 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2437 is superior to 1024 2018-01-13 07:26:07,354 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741873_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:07,354 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741873_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:07,357 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:26:07,357 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:26:07,360 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:07,361 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:07,361 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:07,361 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:07,362 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:07,363 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:35806 2018-01-13 07:26:07,363 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,364 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,365 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,367 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,368 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:35806 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:07,373 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:358060x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:07,375 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:35806-0x160ee689450003a connected 2018-01-13 07:26:07,440 DEBUG [main] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:07,441 DEBUG [main] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:07,441 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=35806 2018-01-13 07:26:07,442 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=35806 2018-01-13 07:26:07,443 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=35806 2018-01-13 07:26:07,443 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:07,445 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:07,445 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:07,446 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:07,446 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:07,446 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:07,447 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:55974 2018-01-13 07:26:07,448 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,448 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,449 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,450 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,452 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:55974 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:07,457 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:559740x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:07,458 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:55974-0x160ee689450003b connected 2018-01-13 07:26:07,488 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:26:07,491 DEBUG [main] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:07,491 DEBUG [main] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:07,492 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=55974 2018-01-13 07:26:07,492 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=55974 2018-01-13 07:26:07,493 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=55974 2018-01-13 07:26:07,494 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:07,553 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:07,553 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:07,553 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:07,554 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:07,554 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:07,554 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:07,555 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:51604 2018-01-13 07:26:07,555 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,556 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,557 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,559 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,560 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:51604 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:07,730 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:516040x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:07,732 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:51604-0x160ee689450003c connected 2018-01-13 07:26:07,732 DEBUG [main] zookeeper.ZKUtil(357): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:07,732 DEBUG [main] zookeeper.ZKUtil(357): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:07,733 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=51604 2018-01-13 07:26:07,734 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=51604 2018-01-13 07:26:07,735 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=51604 2018-01-13 07:26:07,759 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:07,760 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:07,760 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:07,760 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:07,760 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:07,760 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:07,761 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:51707 2018-01-13 07:26:07,762 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,762 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,764 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,766 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,767 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:51707 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:07,789 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:517070x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:07,790 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:51707-0x160ee689450003d connected 2018-01-13 07:26:07,790 DEBUG [main] zookeeper.ZKUtil(357): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:07,791 DEBUG [main] zookeeper.ZKUtil(357): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:07,791 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=51707 2018-01-13 07:26:07,792 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=51707 2018-01-13 07:26:07,794 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=51707 2018-01-13 07:26:07,795 INFO [M:0;asf911:35806] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:07,795 INFO [M:1;asf911:55974] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:07,813 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(355): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:07,814 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:07,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:07,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:07,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:07,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:07,829 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741856_1032, blk_1073741857_1033, blk_1073741858_1034, blk_1073741859_1035, blk_1073741860_1036, blk_1073741861_1037, blk_1073741862_1043, blk_1073741863_1039, blk_1073741864_1040, blk_1073741865_1041, blk_1073741866_1042, blk_1073741868_1045, blk_1073741869_1046, blk_1073741870_1047, blk_1073741872_1049, blk_1073741855_1031] 2018-01-13 07:26:07,832 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:26:07,832 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(355): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:07,834 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:07,837 INFO [M:0;asf911:35806] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,35806,1515828367359 from backup master directory 2018-01-13 07:26:07,837 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:07,837 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:07,837 INFO [M:1;asf911:55974] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,35806,1515828367359; waiting to become the next active master 2018-01-13 07:26:07,837 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:07,837 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:07,847 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:07,847 WARN [M:0;asf911:35806] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:07,847 INFO [M:0;asf911:35806] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:07,884 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741874_1051{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:07,884 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741874_1051{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:07,886 DEBUG [M:0;asf911:35806] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 08a2f168-65be-4809-887f-6428a5b8dd01 2018-01-13 07:26:07,891 INFO [M:0;asf911:35806] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:26:07,892 INFO [M:0;asf911:35806] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:26:07,905 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741875_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:07,915 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741875_1052 size 32 2018-01-13 07:26:07,916 DEBUG [M:0;asf911:35806] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:07,920 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:07,920 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:07,921 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,921 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:07,925 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:07,925 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:07,925 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,926 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:07,928 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:07,929 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:07,929 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,929 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:07,932 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:07,932 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:07,933 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,933 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:07,936 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:07,937 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:07,937 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:07,937 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:07,939 DEBUG [M:0;asf911:35806] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:07,941 DEBUG [M:0;asf911:35806] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:07,945 DEBUG [M:0;asf911:35806] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:07,945 INFO [M:0;asf911:35806] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:26:07,945 DEBUG [M:0;asf911:35806] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:07,945 DEBUG [M:0;asf911:35806] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:07,945 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:07,946 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:07,946 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:07,946 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:07,946 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:07,946 INFO [M:0;asf911:35806] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:07,960 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741876_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:07,960 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741876_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:07,963 DEBUG [M:0;asf911:35806] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:07,984 INFO [M:0;asf911:35806] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:07,984 INFO [M:0;asf911:35806] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:07,985 DEBUG [M:0;asf911:35806] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:08,006 INFO [M:0;asf911:35806] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x55ab7062 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:08,016 DEBUG [M:0;asf911:35806] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2dc68479, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:08,044 INFO [M:0;asf911:35806] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:08,045 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:08,045 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:08,057 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:08,058 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:08,058 INFO [M:0;asf911:35806] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:08,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:08,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:08,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:08,065 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:08,065 INFO [M:0;asf911:35806] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,35806,1515828367359, sessionid=0x160ee689450003a, setting cluster-up flag (Was=false) 2018-01-13 07:26:08,115 INFO [M:0;asf911:35806] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:08,116 DEBUG [M:0;asf911:35806] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:08,157 INFO [M:0;asf911:35806] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:08,158 DEBUG [M:0;asf911:35806] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:08,160 INFO [M:0;asf911:35806] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:08,160 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:35806, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:08,161 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:35806, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:08,161 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:35806, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:08,161 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:35806, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:08,161 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:35806, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:08,161 DEBUG [M:0;asf911:35806] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:35806, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:08,163 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:08,163 INFO [M:0;asf911:35806] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:08,172 DEBUG [M:0;asf911:35806] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:26:08,173 INFO [M:0;asf911:35806] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:26:08,173 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 10msec 2018-01-13 07:26:08,173 DEBUG [M:0;asf911:35806] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:26:08,173 DEBUG [M:0;asf911:35806] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:26:08,173 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:26:08,180 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828398180 2018-01-13 07:26:08,180 INFO [M:0;asf911:35806] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:08,180 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:08,181 INFO [M:0;asf911:35806] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:08,182 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:08,182 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:08,182 INFO [M:0;asf911:35806] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:08,184 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:08,185 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:08,185 DEBUG [M:0;asf911:35806] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:08,185 DEBUG [M:0;asf911:35806] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:35806-HFileCleaner.large.0-1515828368185,5,main] 2018-01-13 07:26:08,185 DEBUG [M:0;asf911:35806] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:35806-HFileCleaner.small.0-1515828368185,5,main] 2018-01-13 07:26:08,185 INFO [M:0;asf911:35806] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828367813, completionTime=-1 2018-01-13 07:26:08,185 INFO [M:0;asf911:35806] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:08,190 DEBUG [M:0;asf911:35806-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:08,192 DEBUG [M:0;asf911:35806-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee689450003f connected 2018-01-13 07:26:08,206 INFO [RS:1;asf911:51707] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x7fc91be3 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:08,206 INFO [RS:0;asf911:51604] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2df414db to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:08,224 DEBUG [RS:0;asf911:51604] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2333ef15, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:08,225 INFO [RS:0;asf911:51604] regionserver.HRegionServer(866): ClusterId : 08a2f168-65be-4809-887f-6428a5b8dd01 2018-01-13 07:26:08,225 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:08,225 DEBUG [RS:1;asf911:51707] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@710eea6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:08,231 DEBUG [RS:0;asf911:51604] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:08,232 INFO [RS:1;asf911:51707] regionserver.HRegionServer(866): ClusterId : 08a2f168-65be-4809-887f-6428a5b8dd01 2018-01-13 07:26:08,232 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:08,232 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:08,233 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:08,240 DEBUG [RS:1;asf911:51707] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:08,248 DEBUG [RS:0;asf911:51604] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:08,249 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:08,249 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:08,249 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:08,249 DEBUG [RS:0;asf911:51604] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@270c7fab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:08,250 DEBUG [RS:0;asf911:51604] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:51604 2018-01-13 07:26:08,250 INFO [RS:0;asf911:51604] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:08,250 INFO [RS:0;asf911:51604] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:08,256 DEBUG [RS:1;asf911:51707] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:08,257 INFO [RS:0;asf911:51604] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,35806,1515828367359 with port=51604, startcode=1515828367517 2018-01-13 07:26:08,257 DEBUG [RS:0;asf911:51604] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:08,257 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:08,258 DEBUG [RS:1;asf911:51707] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75317093, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:08,258 DEBUG [RS:1;asf911:51707] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:51707 2018-01-13 07:26:08,258 INFO [RS:1;asf911:51707] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:08,258 INFO [RS:1;asf911:51707] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:08,259 INFO [RS:1;asf911:51707] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,35806,1515828367359 with port=51707, startcode=1515828367758 2018-01-13 07:26:08,259 DEBUG [RS:1;asf911:51707] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:08,259 INFO [RS-EventLoopGroup-20-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.8 (auth:SIMPLE) 2018-01-13 07:26:08,260 INFO [RS-EventLoopGroup-20-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 51288 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:08,263 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35806] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,263 INFO [RS-EventLoopGroup-20-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.9 (auth:SIMPLE) 2018-01-13 07:26:08,263 INFO [RS-EventLoopGroup-20-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 45040 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:08,264 DEBUG [RS:0;asf911:51604] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:08,264 DEBUG [RS:0;asf911:51604] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:08,264 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35806] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,264 DEBUG [RS:0;asf911:51604] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:08,264 DEBUG [RS:1;asf911:51707] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:08,265 DEBUG [RS:1;asf911:51707] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:08,265 DEBUG [RS:1;asf911:51707] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:08,273 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:08,282 DEBUG [RS:0;asf911:51604] zookeeper.ZKUtil(355): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,282 DEBUG [RS:1;asf911:51707] zookeeper.ZKUtil(355): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,282 WARN [RS:0;asf911:51604] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:08,282 WARN [RS:1;asf911:51707] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:08,282 DEBUG [RS:0;asf911:51604] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,282 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,282 DEBUG [RS:1;asf911:51707] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,283 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,286 INFO [M:0;asf911:35806] master.ServerManager(891): Finished wait on RegionServer count=2; waited=101ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:08,292 INFO [M:0;asf911:35806] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:08,292 INFO [RS:1;asf911:51707] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:08,292 INFO [RS:0;asf911:51604] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:08,293 DEBUG [RS:1;asf911:51707] zookeeper.ZKUtil(355): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,293 DEBUG [RS:0;asf911:51604] zookeeper.ZKUtil(355): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,294 DEBUG [RS:1;asf911:51707] zookeeper.ZKUtil(355): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,294 DEBUG [RS:0;asf911:51604] zookeeper.ZKUtil(355): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,295 INFO [RS:0;asf911:51604] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x02211a61 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:08,295 INFO [RS:1;asf911:51707] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x7f0249c7 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:08,316 DEBUG [RS:1;asf911:51707] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@501c1f1e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:08,316 DEBUG [RS:1;asf911:51707] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:08,316 INFO [RS:1;asf911:51707] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:08,316 DEBUG [RS:0;asf911:51604] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@662f57fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:08,317 DEBUG [RS:0;asf911:51604] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:08,317 INFO [RS:0;asf911:51604] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:08,319 INFO [RS:0;asf911:51604] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:08,319 INFO [RS:1;asf911:51707] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:08,320 INFO [RS:0;asf911:51604] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:08,320 INFO [RS:1;asf911:51707] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:08,320 INFO [RS:0;asf911:51604] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:08,322 INFO [RS:1;asf911:51707] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:08,329 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:51707, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,329 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:51604, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,329 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:51707, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:08,329 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:51604, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:08,330 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:51707, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,330 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:51604, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,330 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:51707, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,330 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:51604, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,330 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:51707, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:08,330 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:51707, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:08,330 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:51604, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:08,331 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:51604, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:08,331 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:51604, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:08,331 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:51707, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:08,331 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:51604, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,331 DEBUG [RS:0;asf911:51604] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:51604, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:08,331 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:51707, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:08,331 DEBUG [RS:1;asf911:51707] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:51707, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:08,356 INFO [RS:0;asf911:51604] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:08,356 INFO [SplitLogWorker-asf911:51604] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,51604,1515828367517 starting 2018-01-13 07:26:08,356 INFO [SplitLogWorker-asf911:51707] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,51707,1515828367758 starting 2018-01-13 07:26:08,356 INFO [RS:1;asf911:51707] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:08,377 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,51707,1515828367758, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:51707, sessionid=0x160ee689450003d 2018-01-13 07:26:08,377 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:08,377 DEBUG [RS:1;asf911:51707] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,377 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,51707,1515828367758' 2018-01-13 07:26:08,377 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:08,378 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:08,378 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:08,379 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:08,379 DEBUG [RS:1;asf911:51707] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:08,379 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,51707,1515828367758' 2018-01-13 07:26:08,379 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:08,380 DEBUG [RS:1;asf911:51707] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:08,380 DEBUG [RS:1;asf911:51707] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:08,380 INFO [RS:1;asf911:51707] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:08,380 INFO [RS:1;asf911:51707] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:08,383 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,51604,1515828367517, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:51604, sessionid=0x160ee689450003c 2018-01-13 07:26:08,383 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:08,383 DEBUG [RS:0;asf911:51604] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,383 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,51604,1515828367517' 2018-01-13 07:26:08,383 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:08,384 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:08,384 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:08,385 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:08,385 DEBUG [RS:0;asf911:51604] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:08,385 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,51604,1515828367517' 2018-01-13 07:26:08,385 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:08,385 DEBUG [RS:0;asf911:51604] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:08,386 DEBUG [RS:0;asf911:51604] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:08,386 INFO [RS:0;asf911:51604] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:08,386 INFO [RS:0;asf911:51604] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:08,464 DEBUG [M:0;asf911:35806] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:08,466 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:08,466 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:08,467 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:08,469 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:26:08,511 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:08,512 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:08,512 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:26:08,512 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:26:08,553 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:08,553 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:26:08,704 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:26:08,704 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:26:08,712 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,385 INFO [RS:1;asf911:51707] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C51707%2C1515828367758, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:09,392 INFO [RS:0;asf911:51604] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C51604%2C1515828367517, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:09,395 DEBUG [RS-EventLoopGroup-23-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:09,395 DEBUG [RS-EventLoopGroup-23-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:09,400 INFO [RS:1;asf911:51707] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758/asf911.gq1.ygridcore.net%2C51707%2C1515828367758.1515828369386 2018-01-13 07:26:09,400 DEBUG [RS:1;asf911:51707] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:09,405 DEBUG [RS-EventLoopGroup-23-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:09,405 DEBUG [RS-EventLoopGroup-23-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:09,408 INFO [RS:0;asf911:51604] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517/asf911.gq1.ygridcore.net%2C51604%2C1515828367517.1515828369394 2018-01-13 07:26:09,408 DEBUG [RS:0;asf911:51604] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:09,411 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,432 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:26:09,464 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,616 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,51604,1515828367517 version=3145728 2018-01-13 07:26:09,617 DEBUG [RSProcedureDispatcher-pool38-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,618 DEBUG [RSProcedureDispatcher-pool38-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:09,621 INFO [RS-EventLoopGroup-22-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:09,622 INFO [RS-EventLoopGroup-22-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48578 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:09,625 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=51604] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:09,626 INFO [RS_OPEN_META-asf911:51604-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:09,632 INFO [RS_OPEN_META-asf911:51604-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C51604%2C1515828367517.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:09,653 DEBUG [RS-EventLoopGroup-23-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:09,654 DEBUG [RS-EventLoopGroup-23-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:09,674 INFO [RS_OPEN_META-asf911:51604-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517/asf911.gq1.ygridcore.net%2C51604%2C1515828367517.meta.1515828369633.meta 2018-01-13 07:26:09,674 DEBUG [RS_OPEN_META-asf911:51604-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:09,675 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:09,675 DEBUG [RS_OPEN_META-asf911:51604-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:09,675 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:09,675 INFO [RS_OPEN_META-asf911:51604-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:09,676 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:09,676 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:09,679 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:09,679 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:09,680 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:09,680 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:09,680 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:09,682 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:09,682 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:09,682 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:09,683 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:09,683 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:09,685 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:09,685 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:09,685 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:09,685 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:09,686 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:09,687 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:09,687 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:09,688 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:09,688 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:09,688 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:09,690 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:09,690 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:09,690 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:09,690 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:09,691 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:09,693 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:09,696 DEBUG [RS_OPEN_META-asf911:51604-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:09,699 DEBUG [RS_OPEN_META-asf911:51604-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:09,700 INFO [RS_OPEN_META-asf911:51604-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:26:09,715 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:09,722 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=35806] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,723 DEBUG [ProcExecWrkr-6] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828369723}]},"ts":1515828369723} 2018-01-13 07:26:09,723 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:09,731 DEBUG [RS_OPEN_META-asf911:51604-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,734 DEBUG [hconnection-0x58d714d6-shared-pool41-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:09,737 INFO [RS-EventLoopGroup-22-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:09,737 INFO [RS-EventLoopGroup-22-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48581 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:09,745 INFO [ProcExecWrkr-6] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:09,750 INFO [ProcExecWrkr-6] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:09,772 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:09,845 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:26:09,928 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:09,928 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.2600sec 2018-01-13 07:26:10,074 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.6360sec 2018-01-13 07:26:10,074 INFO [M:0;asf911:35806] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:10,087 INFO [M:0;asf911:35806] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517 belongs to an existing region server 2018-01-13 07:26:10,088 INFO [M:0;asf911:35806] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758 belongs to an existing region server 2018-01-13 07:26:10,088 INFO [M:0;asf911:35806] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:10,093 INFO [M:0;asf911:35806] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:10,093 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828375093 2018-01-13 07:26:10,093 INFO [M:0;asf911:35806] assignment.AssignmentManager(1219): Joined the cluster in 5msec, failover=true 2018-01-13 07:26:10,100 INFO [M:0;asf911:35806] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:26:10,100 INFO [M:0;asf911:35806] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:26:10,257 DEBUG [M:0;asf911:35806] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:10,259 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:10,328 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:10,358 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741881_1058{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:10,358 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741881_1058{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:10,361 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:10,363 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:26:10,376 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741882_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:10,376 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741882_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:10,377 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,378 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.: disabling compactions & flushes 2018-01-13 07:26:10,378 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,378 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,458 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:10,460 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:10,467 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:10,515 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:10,516 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828370515}]},"ts":1515828370515} 2018-01-13 07:26:10,520 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:26:10,520 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,526 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517}] 2018-01-13 07:26:10,586 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517 hbase:namespace hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,589 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OFFLINE, location=asf911.gq1.ygridcore.net,51604,1515828367517; forceNewPlan=false, retain=false 2018-01-13 07:26:10,742 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:10,745 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:10,747 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x09f32e9f to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:10,767 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dccab79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:10,769 DEBUG [MultiHConnection-shared--pool42-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:10,772 INFO [RS-EventLoopGroup-22-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:10,772 INFO [RS-EventLoopGroup-22-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48596 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:10,776 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:10,830 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741856_1032, blk_1073741857_1033, blk_1073741858_1034, blk_1073741859_1035, blk_1073741860_1036, blk_1073741861_1037, blk_1073741862_1043, blk_1073741863_1039, blk_1073741864_1040, blk_1073741865_1041, blk_1073741866_1042, blk_1073741868_1045, blk_1073741869_1046, blk_1073741870_1047, blk_1073741872_1049, blk_1073741855_1031] 2018-01-13 07:26:10,926 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,51604,1515828367517 version=3145728 2018-01-13 07:26:10,932 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=51604] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,938 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] regionserver.HRegion(6949): Opening region: {ENCODED => 7d28929f437c47936e45b07fd599478f, NAME => 'hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:10,939 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 7d28929f437c47936e45b07fd599478f 2018-01-13 07:26:10,939 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,945 DEBUG [StoreOpener-7d28929f437c47936e45b07fd599478f-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/7d28929f437c47936e45b07fd599478f/info 2018-01-13 07:26:10,945 DEBUG [StoreOpener-7d28929f437c47936e45b07fd599478f-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/7d28929f437c47936e45b07fd599478f/info 2018-01-13 07:26:10,945 INFO [StoreOpener-7d28929f437c47936e45b07fd599478f-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:10,946 INFO [StoreOpener-7d28929f437c47936e45b07fd599478f-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:10,946 INFO [StoreOpener-7d28929f437c47936e45b07fd599478f-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:10,948 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/7d28929f437c47936e45b07fd599478f 2018-01-13 07:26:10,956 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/7d28929f437c47936e45b07fd599478f/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:10,956 INFO [RS_OPEN_PRIORITY_REGION-asf911:51604-0] regionserver.HRegion(960): Onlined 7d28929f437c47936e45b07fd599478f; next sequenceid=2 2018-01-13 07:26:10,959 INFO [PostOpenDeployTasks:7d28929f437c47936e45b07fd599478f] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,962 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=35806] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:10,963 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:10,963 DEBUG [PostOpenDeployTasks:7d28929f437c47936e45b07fd599478f] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:10,966 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:51604-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. on asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:11,049 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:11,099 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:26:11,099 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:11,099 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=7d28929f437c47936e45b07fd599478f, target=asf911.gq1.ygridcore.net,51604,1515828367517 in 462msec 2018-01-13 07:26:11,100 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828371099}]},"ts":1515828371099} 2018-01-13 07:26:11,105 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:26:11,177 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:11,189 DEBUG [M:0;asf911:35806] zookeeper.ZKUtil(357): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:26:11,238 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.0760sec 2018-01-13 07:26:11,245 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:11,409 DEBUG [M:0;asf911:35806] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,411 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,467 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,527 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,562 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:11,585 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,607 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:11,709 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:11,768 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 455msec 2018-01-13 07:26:11,926 DEBUG [M:0;asf911:35806] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:11,927 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:11,976 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:12,051 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:12,100 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:12,115 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:12,184 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:12,242 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 409msec 2018-01-13 07:26:12,265 DEBUG [M:0;asf911:35806] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:12,273 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:12,282 DEBUG [M:0;asf911:35806] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:12,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:12,290 INFO [M:0;asf911:35806] master.HMaster(928): Master has completed initialization 4.443sec 2018-01-13 07:26:12,291 INFO [M:0;asf911:35806] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:12,291 INFO [M:0;asf911:35806] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:12,292 INFO [M:0;asf911:35806] regionserver.HRegionServer(866): ClusterId : 08a2f168-65be-4809-887f-6428a5b8dd01 2018-01-13 07:26:12,305 DEBUG [hconnection-0x51a18b21-shared-pool12-t5] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:26:12,306 DEBUG [Default-IPC-NioEventLoopGroup-7-11] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:58799 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:58799 2018-01-13 07:26:12,411 DEBUG [hconnection-0x51a18b21-shared-pool12-t5] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:12,436 INFO [RS-EventLoopGroup-22-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:12,437 INFO [RS-EventLoopGroup-22-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48604 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:12,442 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:12,454 INFO [Time-limited test] hbase.TestZooKeeper(135): Starting testMasterSessionExpired 2018-01-13 07:26:12,460 INFO [Time-limited test-EventThread] hbase.HBaseTestingUtility$2(2686): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2018-01-13 07:26:12,560 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-01-13 07:26:12,560 DEBUG [main-EventThread] zookeeper.ZKWatcher(528): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-01-13 07:26:12,560 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-01-13 07:26:12,560 DEBUG [main-EventThread] zookeeper.ZKWatcher(528): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-01-13 07:26:12,560 INFO [Time-limited test] hbase.HBaseTestingUtility(2702): ZK Closed Session 0x160ee689450003a 2018-01-13 07:26:12,574 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,574 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,579 INFO [Time-limited test] hbase.TestZooKeeper(165): Creating table testMasterSessionExpired_1515828372574 2018-01-13 07:26:12,580 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:12,581 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,581 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:12,581 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:12,581 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:12,582 DEBUG [main-EventThread] master.ActiveMasterManager(132): No master available. Notifying waiting threads 2018-01-13 07:26:12,582 DEBUG [Default-IPC-NioEventLoopGroup-7-13] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:42210 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:12,582 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:12,582 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:12,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,590 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:12,591 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:55974 2018-01-13 07:26:12,591 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:12,592 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:12,592 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:12,592 INFO [M:1;asf911:55974] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,55974,1515828367444 from backup master directory 2018-01-13 07:26:12,593 INFO [RS-EventLoopGroup-21-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:12,594 INFO [RS-EventLoopGroup-21-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 44954 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:12,598 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:12,598 WARN [M:1;asf911:55974] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:12,599 INFO [M:1;asf911:55974] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:12,616 DEBUG [M:1;asf911:55974] util.FSTableDescriptors(787): Current tableInfoPath = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:12,620 DEBUG [M:1;asf911:55974] util.FSTableDescriptors(791): TableInfo already exists.. Skipping creation 2018-01-13 07:26:12,650 INFO [M:1;asf911:55974] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:12,651 INFO [M:1;asf911:55974] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:12,652 DEBUG [M:1;asf911:55974] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:12,673 INFO [M:1;asf911:55974] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x347d619c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:12,683 DEBUG [M:1;asf911:55974] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64fa0e0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:12,697 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:42210 this server is in the failed servers list 2018-01-13 07:26:12,697 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterSessionExpired(TestZooKeeper.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:12,704 INFO [M:1;asf911:55974] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:12,705 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:12,705 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:12,706 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:12,706 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:12,706 INFO [M:1;asf911:55974] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:12,708 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:12,708 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:12,713 INFO [M:1;asf911:55974] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,55974,1515828367444, sessionid=0x160ee689450003b, setting cluster-up flag (Was=true) 2018-01-13 07:26:12,755 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:12,756 INFO [M:1;asf911:55974] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:12,757 DEBUG [M:1;asf911:55974] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:12,780 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:12,781 INFO [M:1;asf911:55974] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:12,782 DEBUG [M:1;asf911:55974] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:12,783 INFO [M:1;asf911:55974] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:12,783 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:55974, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:12,783 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:55974, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:12,784 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:55974, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:12,784 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:55974, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:12,784 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:55974, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:12,784 DEBUG [M:1;asf911:55974] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:55974, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:12,786 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:12,786 INFO [M:1;asf911:55974] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:12,788 INFO [M:1;asf911:55974] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:26:12,788 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockInfoContiguousUnderConstruction(331): BLOCK* blk_1073741877_1054{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} recovery started, primary=ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW] 2018-01-13 07:26:12,789 WARN [IPC Server handler 1 on 45316] namenode.FSNamesystem(4102): DIR* NameSystem.internalReleaseLease: File /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log has not been closed. Lease recovery is in progress. RecoveryId = 1060 for block blk_1073741877_1054{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} 2018-01-13 07:26:12,789 INFO [M:1;asf911:55974] util.FSHDFSUtils(284): Failed to recover lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-01-13 07:26:12,789 DEBUG [M:1;asf911:55974] procedure.MasterProcedureEnv$WALStoreLeaseRecovery$1(66): Recover Procedure Store log lease: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:26:12,900 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:42210 this server is in the failed servers list 2018-01-13 07:26:12,900 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterSessionExpired(TestZooKeeper.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:13,203 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:42210 this server is in the failed servers list 2018-01-13 07:26:13,204 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterSessionExpired(TestZooKeeper.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:13,711 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:42210 this server is in the failed servers list 2018-01-13 07:26:13,711 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterSessionExpired(TestZooKeeper.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:13,719 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-01-13 07:26:13,719 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:13,719 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase master:35806-0x160ee689450003a received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:13,720 INFO [main-EventThread] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,35806,1515828367359' ***** 2018-01-13 07:26:13,720 INFO [main-EventThread] regionserver.HRegionServer(2172): STOPPED: Stopped by main-EventThread 2018-01-13 07:26:13,720 WARN [main-EventThread] zookeeper.ZKUtil(618): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:13,720 DEBUG [M:0;asf911:35806] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c780715, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:13,720 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:13,720 INFO [M:0;asf911:35806] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,35806,1515828367359 2018-01-13 07:26:13,721 ERROR [main-EventThread] master.ActiveMasterManager(280): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.connectionEvent(ZKWatcher.java:537) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:469) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:13,721 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Expired, path=null 2018-01-13 07:26:13,721 DEBUG [M:0;asf911:35806] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:13,723 INFO [M:0;asf911:35806] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x55ab7062 to localhost:59094 2018-01-13 07:26:13,723 DEBUG [M:0;asf911:35806] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:13,724 INFO [M:0;asf911:35806] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,35806,1515828367359; all regions closed. 2018-01-13 07:26:13,724 DEBUG [M:0;asf911:35806] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:13,724 INFO [M:0;asf911:35806] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,35806,1515828367359 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,35806,1515828367359-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35806,1515828367359-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:35806 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35806,1515828367359-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35806,1515828367359-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35806,1515828367359-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:13,724 INFO [M:0;asf911:35806] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:13,724 INFO [M:0;asf911:35806] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:13,724 DEBUG [M:0;asf911:35806] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:13,725 WARN [M:0;asf911:35806] zookeeper.ZKUtil(618): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:13,725 ERROR [M:0;asf911:35806] zookeeper.ZKWatcher(578): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:13,725 ERROR [M:0;asf911:35806] master.ActiveMasterManager(280): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:13,725 INFO [M:0;asf911:35806] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:13,725 INFO [M:0;asf911:35806] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x09f32e9f to localhost:59094 2018-01-13 07:26:13,726 DEBUG [M:0;asf911:35806] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:13,727 INFO [M:0;asf911:35806] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:13,727 INFO [M:0;asf911:35806] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:13,729 INFO [M:0;asf911:35806] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:13,729 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,730 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,731 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,731 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,733 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,733 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,742 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,743 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,742 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,744 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:13,751 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741877_1054{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:13,752 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741877_1054{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:13,753 INFO [M:0;asf911:35806] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,35806,1515828367359_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:13,753 INFO [M:0;asf911:35806] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:13,753 INFO [M:0;asf911:35806] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:13,756 WARN [M:0;asf911:35806] regionserver.HRegionServer(1146): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf911.gq1.ygridcore.net,35806,1515828367359 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1591) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1143) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:13,757 INFO [M:0;asf911:35806] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,35806,1515828367359; zookeeper connection closed. 2018-01-13 07:26:13,757 INFO [M:0;asf911:35806] regionserver.HRegionServer(1157): M:0;asf911:35806 exiting 2018-01-13 07:26:14,319 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:14,319 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:14,413 DEBUG [RS:1;asf911:51707] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:14,416 DEBUG [RS-EventLoopGroup-23-19] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:35806 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:14,424 DEBUG [RS:0;asf911:51604] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:14,425 DEBUG [RS-EventLoopGroup-23-20] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:35806 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:35806 2018-01-13 07:26:14,721 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:14,723 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:14,723 DEBUG [Default-IPC-NioEventLoopGroup-7-15] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:42210 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:15,417 DEBUG [RS:1;asf911:51707] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:55974 2018-01-13 07:26:15,419 INFO [RS-EventLoopGroup-21-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.9 (auth:SIMPLE) 2018-01-13 07:26:15,420 INFO [RS-EventLoopGroup-21-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 53700 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:15,427 DEBUG [RS:0;asf911:51604] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:55974 2018-01-13 07:26:15,431 INFO [RS-EventLoopGroup-21-4] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.8 (auth:SIMPLE) 2018-01-13 07:26:15,431 INFO [Block report processor] blockmanagement.CorruptReplicasMap(76): BLOCK NameSystem.addToCorruptReplicasMap: blk_1073741877 added as corrupt on 127.0.0.1:49352 by null because block is COMPLETE and reported genstamp 1060 does not match genstamp in block map 1054 2018-01-13 07:26:15,431 INFO [RS-EventLoopGroup-21-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56665 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:15,432 INFO [Block report processor] blockmanagement.CorruptReplicasMap(76): BLOCK NameSystem.addToCorruptReplicasMap: blk_1073741877 added as corrupt on 127.0.0.1:59189 by null because block is COMPLETE and reported genstamp 1060 does not match genstamp in block map 1054 2018-01-13 07:26:16,741 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:16,743 DEBUG [Default-IPC-NioEventLoopGroup-7-16] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:42210 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:16,743 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:16,756 DEBUG [Time-limited test] client.RpcRetryingCallerImpl(132): Call exception, tries=6, retries=6, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2695) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1089) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) , details=, exception=org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2695) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1089) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1156) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1216) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterSessionExpired(TestZooKeeper.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2695) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1089) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:93) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:83) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:363) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:351) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionImplementation.java:1106) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStubNoRetries(ConnectionImplementation.java:1134) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1150) ... 22 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2695) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:1089) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:161) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:191) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-01-13 07:26:16,791 INFO [M:1;asf911:55974] util.FSHDFSUtils(284): Recovered lease, attempt=1 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log after 4003ms 2018-01-13 07:26:16,791 DEBUG [M:1;asf911:55974] wal.WALProcedureStore(1286): Opening Pv2 FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=214; replication=2; blocksize=134217728; modification_time=1515828368171; access_time=1515828368171; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-01-13 07:26:16,795 WARN [M:1;asf911:55974] wal.WALProcedureStore(1305): Unable to read tracker for hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log - Invalid Trailer version. got 116 expected 1 2018-01-13 07:26:16,798 DEBUG [M:1;asf911:55974] wal.WALProcedureStore(1071): Roll new state log: 2 2018-01-13 07:26:16,799 INFO [M:1;asf911:55974] wal.WALProcedureStore(390): Lease acquired for flushLogId: 2 2018-01-13 07:26:16,799 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 4.0130sec 2018-01-13 07:26:16,808 INFO [M:1;asf911:55974] wal.ProcedureWALFormatReader(140): Rebuilding tracker for hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:26:16,815 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor$1(317): Load max pid=6 2018-01-13 07:26:16,817 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor(356): Completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:16,817 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor(356): Completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:16,818 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor(356): Completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:16,818 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor(356): Completed pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:16,818 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 19msec 2018-01-13 07:26:16,820 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828406820 2018-01-13 07:26:16,820 INFO [M:1;asf911:55974] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:16,820 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:16,820 INFO [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:16,821 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:16,821 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:16,821 INFO [M:1;asf911:55974] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:16,823 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:16,824 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:16,824 DEBUG [M:1;asf911:55974] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:16,824 DEBUG [M:1;asf911:55974] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:55974-HFileCleaner.large.0-1515828376824,5,main] 2018-01-13 07:26:16,824 DEBUG [M:1;asf911:55974] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:55974-HFileCleaner.small.0-1515828376824,5,main] 2018-01-13 07:26:16,825 INFO [M:1;asf911:55974] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828367814, completionTime=-1 2018-01-13 07:26:16,825 INFO [M:1;asf911:55974] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:16,832 DEBUG [M:1;asf911:55974-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:16,833 DEBUG [M:1;asf911:55974-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500046 connected 2018-01-13 07:26:17,428 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:17,439 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:17,440 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] assignment.AssignmentManager(996): META REPORTED: rit=OFFLINE, location=null, table=hbase:meta, region=1588230740 2018-01-13 07:26:17,440 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] assignment.AssignmentManager(998): META REPORTED but no procedure found (complete?); set location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:17,476 INFO [M:1;asf911:55974] master.ServerManager(891): Finished wait on RegionServer count=2; waited=651ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:17,479 INFO [M:1;asf911:55974] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:17,670 DEBUG [M:1;asf911:55974] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:17,672 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:17,672 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:17,676 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517 belongs to an existing region server 2018-01-13 07:26:17,676 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758 belongs to an existing region server 2018-01-13 07:26:17,732 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:17,733 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:17,734 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517}] 2018-01-13 07:26:18,041 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:18,041 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OFFLINE, location=asf911.gq1.ygridcore.net,51604,1515828367517; forceNewPlan=false, retain=true 2018-01-13 07:26:18,197 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:18,202 INFO [ProcExecWrkr-2] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,207 INFO [ProcExecWrkr-2] assignment.RegionTransitionProcedure(208): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,358 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,51604,1515828367517 version=3145728 2018-01-13 07:26:18,359 DEBUG [RSProcedureDispatcher-pool46-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,360 DEBUG [RSProcedureDispatcher-pool46-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:18,362 INFO [RS-EventLoopGroup-22-6] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:18,362 INFO [RS-EventLoopGroup-22-6] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48669 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:18,362 WARN [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=51604] regionserver.RSRpcServices(1901): Received OPEN for the region:hbase:meta,,1.1588230740, which is already online 2018-01-13 07:26:18,443 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] assignment.AssignmentManager(996): META REPORTED: rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517, table=hbase:meta, region=1588230740 2018-01-13 07:26:18,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=0, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517; rit=OPENING, location=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,444 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828378444}]},"ts":1515828378444} 2018-01-13 07:26:18,450 DEBUG [hconnection-0x5f0e4f53-shared-pool47-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:18,452 INFO [RS-EventLoopGroup-22-7] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:18,453 INFO [RS-EventLoopGroup-22-7] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48671 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:18,456 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:18,457 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,524 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,582 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:18,583 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,51604,1515828367517 in 732msec 2018-01-13 07:26:18,641 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.1030sec 2018-01-13 07:26:18,642 INFO [M:1;asf911:55974] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,651 INFO [M:1;asf911:55974] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51604,1515828367517 belongs to an existing region server 2018-01-13 07:26:18,651 INFO [M:1;asf911:55974] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,51707,1515828367758 belongs to an existing region server 2018-01-13 07:26:18,652 INFO [M:1;asf911:55974] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:18,655 INFO [M:1;asf911:55974] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 7d28929f437c47936e45b07fd599478f, NAME => 'hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.', STARTKEY => '', ENDKEY => ''} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,51604,1515828367517 regionLocation=asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:18,656 INFO [M:1;asf911:55974] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:18,656 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828383656 2018-01-13 07:26:18,656 INFO [M:1;asf911:55974] assignment.AssignmentManager(1219): Joined the cluster in 4msec, failover=true 2018-01-13 07:26:18,661 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(355): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:18,688 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:18,698 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:18,723 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:18,732 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:18,732 INFO [M:1;asf911:55974] master.HMaster(928): Master has completed initialization 6.133sec 2018-01-13 07:26:18,733 INFO [M:1;asf911:55974] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:18,733 INFO [M:1;asf911:55974] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:18,734 INFO [M:1;asf911:55974] regionserver.HRegionServer(866): ClusterId : 08a2f168-65be-4809-887f-6428a5b8dd01 2018-01-13 07:26:20,774 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:20,776 DEBUG [Default-IPC-NioEventLoopGroup-7-17] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:42210 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 2018-01-13 07:26:20,776 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42210 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42210 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:20,786 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.HMaster$3(1743): Client=jenkins//67.195.81.155 create 'testMasterSessionExpired_1515828372574', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-13 07:26:20,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] procedure2.ProcedureExecutor(869): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:20,951 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:21,007 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,026 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741884_1062{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:21,026 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741884_1062{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:21,032 DEBUG [ProcExecWrkr-8] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/default/testMasterSessionExpired_1515828372574/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:21,034 INFO [RegionOpenAndInitThread-testMasterSessionExpired_1515828372574-1] regionserver.HRegion(6772): creating HRegion testMasterSessionExpired_1515828372574 HTD == 'testMasterSessionExpired_1515828372574', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741885_1063{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:21,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741885_1063{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:21,051 DEBUG [RegionOpenAndInitThread-testMasterSessionExpired_1515828372574-1] regionserver.HRegion(807): Instantiated testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,052 DEBUG [RegionOpenAndInitThread-testMasterSessionExpired_1515828372574-1] regionserver.HRegion(1481): Closing testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.: disabling compactions & flushes 2018-01-13 07:26:21,052 DEBUG [RegionOpenAndInitThread-testMasterSessionExpired_1515828372574-1] regionserver.HRegion(1520): Updates disabled for region testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,052 INFO [RegionOpenAndInitThread-testMasterSessionExpired_1515828372574-1] regionserver.HRegion(1633): Closed testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:21,122 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,124 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.","families":{"info":[{"qualifier":"regioninfo","vlen":72,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:21,129 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:21,176 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,177 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMasterSessionExpired_1515828372574","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828381177}]},"ts":1515828381177} 2018-01-13 07:26:21,180 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1786): Updated table testMasterSessionExpired_1515828372574 state to ENABLING in META 2018-01-13 07:26:21,181 DEBUG [ProcExecWrkr-8] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,187 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758}] 2018-01-13 07:26:21,236 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(883): pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758 testMasterSessionExpired_1515828372574 testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,239 INFO [ProcExecWrkr-8] assignment.AssignProcedure(211): Start pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758; rit=OFFLINE, location=asf911.gq1.ygridcore.net,51707,1515828367758; forceNewPlan=false, retain=false 2018-01-13 07:26:21,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:21,391 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:21,395 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,396 INFO [ProcExecWrkr-10] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5a9811ab to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:21,415 DEBUG [ProcExecWrkr-10] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@545fe488, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:21,417 DEBUG [MultiHConnection-shared--pool51-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:21,420 INFO [RS-EventLoopGroup-22-8] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:21,421 INFO [RS-EventLoopGroup-22-8] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48697 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:21,424 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758; rit=OPENING, location=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:21,576 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,51707,1515828367758 version=3145728 2018-01-13 07:26:21,577 DEBUG [RSProcedureDispatcher-pool46-t2] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,578 DEBUG [RSProcedureDispatcher-pool46-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51707 2018-01-13 07:26:21,581 INFO [RS-EventLoopGroup-23-27] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:21,582 INFO [RS-EventLoopGroup-23-27] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 47643 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:21,589 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=51707] regionserver.RSRpcServices(1907): Open testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,600 DEBUG [RS_OPEN_REGION-asf911:51707-0] regionserver.HRegion(6949): Opening region: {ENCODED => cbea968cc5ab65afee33632a0b7fbb17, NAME => 'testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:21,601 DEBUG [RS_OPEN_REGION-asf911:51707-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMasterSessionExpired_1515828372574 cbea968cc5ab65afee33632a0b7fbb17 2018-01-13 07:26:21,601 DEBUG [RS_OPEN_REGION-asf911:51707-0] regionserver.HRegion(807): Instantiated testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,607 DEBUG [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterSessionExpired_1515828372574/cbea968cc5ab65afee33632a0b7fbb17/fam 2018-01-13 07:26:21,607 DEBUG [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterSessionExpired_1515828372574/cbea968cc5ab65afee33632a0b7fbb17/fam 2018-01-13 07:26:21,608 INFO [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:21,608 INFO [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:21,608 INFO [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] hfile.CacheConfig(239): Created cacheConfig for fam: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:21,609 INFO [StoreOpener-cbea968cc5ab65afee33632a0b7fbb17-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:21,610 INFO [RS_OPEN_REGION-asf911:51707-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,611 DEBUG [RS_OPEN_REGION-asf911:51707-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterSessionExpired_1515828372574/cbea968cc5ab65afee33632a0b7fbb17 2018-01-13 07:26:21,614 DEBUG [RS_OPEN_REGION-asf911:51707-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMasterSessionExpired_1515828372574 descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:21,620 DEBUG [RS_OPEN_REGION-asf911:51707-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterSessionExpired_1515828372574/cbea968cc5ab65afee33632a0b7fbb17/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:21,620 INFO [RS_OPEN_REGION-asf911:51707-0] regionserver.HRegion(960): Onlined cbea968cc5ab65afee33632a0b7fbb17; next sequenceid=2 2018-01-13 07:26:21,622 INFO [PostOpenDeployTasks:cbea968cc5ab65afee33632a0b7fbb17] regionserver.HRegionServer(2197): Post open deploy tasks for testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758; rit=OPENING, location=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,627 INFO [ProcExecWrkr-7] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,627 DEBUG [PostOpenDeployTasks:cbea968cc5ab65afee33632a0b7fbb17] regionserver.HRegionServer(2221): Finished post open deploy task for testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:21,630 DEBUG [RS_OPEN_REGION-asf911:51707-0] handler.OpenRegionHandler(128): Opened testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. on asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,682 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:21,723 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterSessionExpired_1515828372574; resume parent processing. 2018-01-13 07:26:21,724 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,724 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMasterSessionExpired_1515828372574, region=cbea968cc5ab65afee33632a0b7fbb17, target=asf911.gq1.ygridcore.net,51707,1515828367758 in 450msec 2018-01-13 07:26:21,724 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMasterSessionExpired_1515828372574","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828381724}]},"ts":1515828381724} 2018-01-13 07:26:21,728 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table testMasterSessionExpired_1515828372574 state to ENABLED in META 2018-01-13 07:26:21,795 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testMasterSessionExpired_1515828372574 2018-01-13 07:26:21,851 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMasterSessionExpired_1515828372574 in 1.0090sec 2018-01-13 07:26:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:22,066 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3638): Operation: CREATE, Table Name: default:testMasterSessionExpired_1515828372574 completed 2018-01-13 07:26:22,067 INFO [Time-limited test] hbase.TestZooKeeper(176): Putting table testMasterSessionExpired_1515828372574 2018-01-13 07:26:22,074 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51707 2018-01-13 07:26:22,076 INFO [RS-EventLoopGroup-23-28] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:22,076 INFO [RS-EventLoopGroup-23-28] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 47648 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:22,177 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,51604,1515828367517: killAll ***** 2018-01-13 07:26:22,178 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:22,219 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 255458928 }, "NonHeapMemoryUsage" : { "committed" : 128581632, "init" : 2555904, "max" : -1, "used" : 126562184 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 40064, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 182, "TotalCallTime_min" : 0, "TotalCallTime_max" : 99, "TotalCallTime_mean" : 5, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 3, "TotalCallTime_95th_percentile" : 12, "TotalCallTime_98th_percentile" : 99, "TotalCallTime_99th_percentile" : 99, "TotalCallTime_99.9th_percentile" : 99, "TotalCallTime_TimeRangeCount_0-1" : 14, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_10-30" : 1, "TotalCallTime_TimeRangeCount_30-100" : 1, "TotalCallTime_TimeRangeCount_600000-inf" : 156, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 182, "ResponseSize_min" : 0, "ResponseSize_max" : 937, "ResponseSize_mean" : 186, "ResponseSize_25th_percentile" : 6, "ResponseSize_median" : 11, "ResponseSize_75th_percentile" : 119, "ResponseSize_90th_percentile" : 769, "ResponseSize_95th_percentile" : 873, "ResponseSize_98th_percentile" : 935, "ResponseSize_99th_percentile" : 936, "ResponseSize_99.9th_percentile" : 936, "ResponseSize_SizeRangeCount_0-10" : 13, "ResponseSize_SizeRangeCount_10-100" : 5, "ResponseSize_SizeRangeCount_100-1000" : 8, "ResponseSize_SizeRangeCount_100000000-inf" : 156, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 182, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 99, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 3, "ProcessCallTime_95th_percentile" : 11, "ProcessCallTime_98th_percentile" : 99, "ProcessCallTime_99th_percentile" : 99, "ProcessCallTime_99.9th_percentile" : 99, "ProcessCallTime_TimeRangeCount_0-1" : 15, "ProcessCallTime_TimeRangeCount_1-3" : 9, "ProcessCallTime_TimeRangeCount_10-30" : 1, "ProcessCallTime_TimeRangeCount_30-100" : 1, "ProcessCallTime_TimeRangeCount_600000-inf" : 156, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 31, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 182, "RequestSize_min" : 70, "RequestSize_max" : 908, "RequestSize_mean" : 178, "RequestSize_25th_percentile" : 88, "RequestSize_median" : 110, "RequestSize_75th_percentile" : 145, "RequestSize_90th_percentile" : 482, "RequestSize_95th_percentile" : 643, "RequestSize_98th_percentile" : 821, "RequestSize_99th_percentile" : 864, "RequestSize_99.9th_percentile" : 903, "RequestSize_SizeRangeCount_0-10" : 1, "RequestSize_SizeRangeCount_10-100" : 9, "RequestSize_SizeRangeCount_100-1000" : 16, "RequestSize_SizeRangeCount_100000000-inf" : 156, "sentBytes" : 18618, "QueueCallTime_num_ops" : 182, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 26, "QueueCallTime_TimeRangeCount_600000-inf" : 156, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:22,222 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,51604,1515828367517 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,51604,1515828367517: killAll ***** 2018-01-13 07:26:22,223 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,51604,1515828367517' ***** 2018-01-13 07:26:22,223 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:22,223 INFO [RS:0;asf911:51604] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:22,223 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,51707,1515828367758: killAll ***** 2018-01-13 07:26:22,224 INFO [SplitLogWorker-asf911:51604] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:22,223 INFO [RS:0;asf911:51604] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:22,224 INFO [SplitLogWorker-asf911:51604] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,51604,1515828367517 exiting 2018-01-13 07:26:22,224 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:22,225 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:22,224 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:22,224 INFO [RS:0;asf911:51604] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:22,226 INFO [RS:0;asf911:51604] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:22,226 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:22,226 DEBUG [RS:0;asf911:51604] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:22,226 DEBUG [RS_CLOSE_REGION-asf911:51604-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:22,226 INFO [RS:0;asf911:51604] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x2df414db to localhost:59094 2018-01-13 07:26:22,228 DEBUG [RS:0;asf911:51604] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,228 DEBUG [RS_CLOSE_REGION-asf911:51604-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.: disabling compactions & flushes 2018-01-13 07:26:22,228 INFO [RS:0;asf911:51604] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:22,228 INFO [RS:0;asf911:51604] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:22,228 INFO [RS:0;asf911:51604] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:22,228 DEBUG [RS_CLOSE_REGION-asf911:51604-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:22,229 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:26:22,229 DEBUG [RS_CLOSE_META-asf911:51604-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:22,229 DEBUG [RS:0;asf911:51604] regionserver.HRegionServer(1392): Online Regions={7d28929f437c47936e45b07fd599478f=hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f., 1588230740=hbase:meta,,1.1588230740} 2018-01-13 07:26:22,230 INFO [StoreCloserThread-hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f.-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:22,229 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 256612440 }, "NonHeapMemoryUsage" : { "committed" : 128581632, "init" : 2555904, "max" : -1, "used" : 126569536 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 40064, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 182, "TotalCallTime_min" : 0, "TotalCallTime_max" : 99, "TotalCallTime_mean" : 5, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 3, "TotalCallTime_95th_percentile" : 12, "TotalCallTime_98th_percentile" : 99, "TotalCallTime_99th_percentile" : 99, "TotalCallTime_99.9th_percentile" : 99, "TotalCallTime_TimeRangeCount_0-1" : 14, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_10-30" : 1, "TotalCallTime_TimeRangeCount_30-100" : 1, "TotalCallTime_TimeRangeCount_600000-inf" : 156, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 182, "ResponseSize_min" : 0, "ResponseSize_max" : 937, "ResponseSize_mean" : 186, "ResponseSize_25th_percentile" : 6, "ResponseSize_median" : 11, "ResponseSize_75th_percentile" : 119, "ResponseSize_90th_percentile" : 769, "ResponseSize_95th_percentile" : 873, "ResponseSize_98th_percentile" : 935, "ResponseSize_99th_percentile" : 936, "ResponseSize_99.9th_percentile" : 936, "ResponseSize_SizeRangeCount_0-10" : 13, "ResponseSize_SizeRangeCount_10-100" : 5, "ResponseSize_SizeRangeCount_100-1000" : 8, "ResponseSize_SizeRangeCount_100000000-inf" : 156, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 182, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 99, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 3, "ProcessCallTime_95th_percentile" : 11, "ProcessCallTime_98th_percentile" : 99, "ProcessCallTime_99th_percentile" : 99, "ProcessCallTime_99.9th_percentile" : 99, "ProcessCallTime_TimeRangeCount_0-1" : 15, "ProcessCallTime_TimeRangeCount_1-3" : 9, "ProcessCallTime_TimeRangeCount_10-30" : 1, "ProcessCallTime_TimeRangeCount_30-100" : 1, "ProcessCallTime_TimeRangeCount_600000-inf" : 156, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 31, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 182, "RequestSize_min" : 70, "RequestSize_max" : 908, "RequestSize_mean" : 178, "RequestSize_25th_percentile" : 88, "RequestSize_median" : 110, "RequestSize_75th_percentile" : 145, "RequestSize_90th_percentile" : 482, "RequestSize_95th_percentile" : 643, "RequestSize_98th_percentile" : 821, "RequestSize_99th_percentile" : 864, "RequestSize_99.9th_percentile" : 903, "RequestSize_SizeRangeCount_0-10" : 1, "RequestSize_SizeRangeCount_10-100" : 9, "RequestSize_SizeRangeCount_100-1000" : 16, "RequestSize_SizeRangeCount_100000000-inf" : 156, "sentBytes" : 18618, "QueueCallTime_num_ops" : 182, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 26, "QueueCallTime_TimeRangeCount_600000-inf" : 156, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:22,230 DEBUG [RS_CLOSE_META-asf911:51604-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:22,233 DEBUG [RS_CLOSE_META-asf911:51604-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:22,231 ERROR [RS_CLOSE_REGION-asf911:51604-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:26:22,233 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55974] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,51707,1515828367758 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,51707,1515828367758: killAll ***** 2018-01-13 07:26:22,235 INFO [RS_CLOSE_REGION-asf911:51604-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:22,235 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:22,235 DEBUG [RS_CLOSE_REGION-asf911:51604-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828370100.7d28929f437c47936e45b07fd599478f. 2018-01-13 07:26:22,235 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,51707,1515828367758' ***** 2018-01-13 07:26:22,236 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:22,235 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:22,236 INFO [RS:1;asf911:51707] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:22,236 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:22,236 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:22,236 INFO [SplitLogWorker-asf911:51707] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:22,236 INFO [RS:1;asf911:51707] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:22,236 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:22,237 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:22,239 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:22,237 INFO [RS:1;asf911:51707] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:22,237 INFO [SplitLogWorker-asf911:51707] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,51707,1515828367758 exiting 2018-01-13 07:26:22,237 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,55974,1515828367444' ***** 2018-01-13 07:26:22,239 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:22,239 INFO [RS:1;asf911:51707] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:22,239 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:22,242 DEBUG [RS_CLOSE_REGION-asf911:51707-0] handler.CloseRegionHandler(92): Processing close of testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:22,239 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:22,242 DEBUG [M:1;asf911:55974] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32ead559, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:22,244 INFO [M:1;asf911:55974] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:22,244 DEBUG [M:1;asf911:55974] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:22,242 ERROR [RS_CLOSE_META-asf911:51604-0] regionserver.HRegion(1620): Memstore size is 3029 2018-01-13 07:26:22,244 DEBUG [RS_CLOSE_META-asf911:51604-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:22,242 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:22,246 DEBUG [RS:1;asf911:51707] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:22,246 INFO [RS_CLOSE_META-asf911:51604-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:22,244 INFO [M:1;asf911:55974] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x347d619c to localhost:59094 2018-01-13 07:26:22,243 DEBUG [RS_CLOSE_REGION-asf911:51707-0] regionserver.HRegion(1481): Closing testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.: disabling compactions & flushes 2018-01-13 07:26:22,246 DEBUG [RS_CLOSE_REGION-asf911:51707-0] regionserver.HRegion(1520): Updates disabled for region testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:22,246 DEBUG [M:1;asf911:55974] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,246 DEBUG [RS_CLOSE_META-asf911:51604-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:22,246 INFO [RS:1;asf911:51707] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x7fc91be3 to localhost:59094 2018-01-13 07:26:22,250 DEBUG [RS:1;asf911:51707] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,250 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:22,250 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:22,250 INFO [M:1;asf911:55974] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,55974,1515828367444; all regions closed. 2018-01-13 07:26:22,252 DEBUG [M:1;asf911:55974] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,250 INFO [StoreCloserThread-testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.-1] regionserver.HStore(930): Closed fam 2018-01-13 07:26:22,252 INFO [M:1;asf911:55974] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,55974,1515828367444 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,55974,1515828367444-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55974,1515828367444-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55974,1515828367444-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:55974 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55974,1515828367444-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55974,1515828367444-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:22,251 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1388): Waiting on 1 regions to close 2018-01-13 07:26:22,252 DEBUG [RS:1;asf911:51707] regionserver.HRegionServer(1392): Online Regions={cbea968cc5ab65afee33632a0b7fbb17=testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17.} 2018-01-13 07:26:22,251 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:26:22,250 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:22,253 WARN [main] zookeeper.ZKUtil(618): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:22,254 ERROR [main] zookeeper.ZKWatcher(578): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:22,254 ERROR [main] master.ActiveMasterManager(280): master:35806-0x160ee689450003a, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:22,255 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,55974,1515828367444 2018-01-13 07:26:22,252 INFO [M:1;asf911:55974] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:22,255 INFO [M:1;asf911:55974] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:22,255 DEBUG [M:1;asf911:55974] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:22,252 ERROR [RS_CLOSE_REGION-asf911:51707-0] regionserver.HRegion(1620): Memstore size is 41 2018-01-13 07:26:22,256 INFO [RS_CLOSE_REGION-asf911:51707-0] regionserver.HRegion(1633): Closed testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:22,256 DEBUG [RS_CLOSE_REGION-asf911:51707-0] handler.CloseRegionHandler(125): Closed testMasterSessionExpired_1515828372574,,1515828380784.cbea968cc5ab65afee33632a0b7fbb17. 2018-01-13 07:26:22,259 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,265 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,266 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:22,267 DEBUG [M:1;asf911:55974] zookeeper.ZKUtil(614): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:22,267 WARN [M:1;asf911:55974] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,267 INFO [M:1;asf911:55974] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:55974-0x160ee689450003b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,267 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,269 INFO [M:1;asf911:55974] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5a9811ab to localhost:59094 2018-01-13 07:26:22,269 DEBUG [M:1;asf911:55974] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,269 INFO [M:1;asf911:55974] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:22,270 INFO [M:1;asf911:55974] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:22,271 INFO [M:1;asf911:55974] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:22,271 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,272 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,272 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,275 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,272 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,272 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,272 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,278 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,278 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,278 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,278 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,278 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,275 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,275 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,275 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,275 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:22,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741883_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:22,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741883_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:22,306 INFO [M:1;asf911:55974] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,55974,1515828367444_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:22,306 INFO [M:1;asf911:55974] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:22,306 INFO [M:1;asf911:55974] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:22,321 DEBUG [M:1;asf911:55974] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,55974,1515828367444 already deleted, retry=false 2018-01-13 07:26:22,332 INFO [M:1;asf911:55974] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,55974,1515828367444; zookeeper connection closed. 2018-01-13 07:26:22,332 INFO [M:1;asf911:55974] regionserver.HRegionServer(1157): M:1;asf911:55974 exiting 2018-01-13 07:26:22,334 INFO [asf911.gq1.ygridcore.net,51707,1515828367758_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 07:26:22,335 INFO [asf911.gq1.ygridcore.net,51604,1515828367517_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 07:26:22,336 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:22,336 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:22,336 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:22,336 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:22,431 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,51604,1515828367517; all regions closed. 2018-01-13 07:26:22,439 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741880_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:22,439 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741880_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:22,449 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741879_1056{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:22,449 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741879_1056{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:22,451 DEBUG [RS:0;asf911:51604] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,452 INFO [RS:0;asf911:51604] regionserver.Leases(147): RS:0;asf911:51604 closing leases 2018-01-13 07:26:22,452 INFO [RS:0;asf911:51604] regionserver.Leases(150): RS:0;asf911:51604 closed leases 2018-01-13 07:26:22,452 INFO [RS:0;asf911:51604] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,51604,1515828367517 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,51604,1515828367517 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:22,452 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:22,452 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,51707,1515828367758; all regions closed. 2018-01-13 07:26:22,453 INFO [RS:0;asf911:51604] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:22,458 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741878_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:22,458 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741878_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:22,458 DEBUG [RS:1;asf911:51707] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:22,459 INFO [RS:1;asf911:51707] regionserver.Leases(147): RS:1;asf911:51707 closing leases 2018-01-13 07:26:22,459 INFO [RS:1;asf911:51707] regionserver.Leases(150): RS:1;asf911:51707 closed leases 2018-01-13 07:26:22,459 INFO [RS:1;asf911:51707] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,51707,1515828367758 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,51707,1515828367758 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:22,459 INFO [RS:1;asf911:51707] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:22,459 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:22,460 INFO [RS:1;asf911:51707] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:22,460 INFO [RS:1;asf911:51707] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:22,460 INFO [RS:1;asf911:51707] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:22,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:22,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,51604,1515828367517 2018-01-13 07:26:22,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:22,466 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:22,473 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51604-0x160ee689450003c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:22,473 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,51604,1515828367517; zookeeper connection closed. 2018-01-13 07:26:22,473 INFO [RS:0;asf911:51604] regionserver.HRegionServer(1157): RS:0;asf911:51604 exiting 2018-01-13 07:26:22,473 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:51707-0x160ee689450003d, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,51707,1515828367758 2018-01-13 07:26:22,474 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@16a44426] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@16a44426 2018-01-13 07:26:22,482 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,51707,1515828367758; zookeeper connection closed. 2018-01-13 07:26:22,482 INFO [RS:1;asf911:51707] regionserver.HRegionServer(1157): RS:1;asf911:51707 exiting 2018-01-13 07:26:22,482 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7111252b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7111252b 2018-01-13 07:26:22,482 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:26:22,491 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741877_1054 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741883_1061 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741879_1056 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741880_1057 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741878_1055 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741884_1062 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741885_1063 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,493 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741876_1053 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:22,494 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741875_1052 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,494 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741881_1058 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,494 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741882_1059 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,494 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741874_1051 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,494 INFO [IPC Server handler 4 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741873_1050 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:22,496 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:22,507 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:22,508 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500048 connected 2018-01-13 07:26:22,509 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:26:22,509 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:26:22,510 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:26:22,510 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:26:22,510 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:26:22,511 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:26:22,511 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:26:22,511 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:26:22,512 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:26:22,512 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:26:22,512 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:22,513 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:26:22,513 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:26:22,515 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:26:22,515 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:26:22,516 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:26:22,517 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:26:22,517 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:26:22,517 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:26:22,518 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:26:22,519 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:26:22,532 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:26:22,533 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:22,538 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500048, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:26:22,573 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testMasterSessionExpired Thread=424 (was 346) Potentially hanging thread: RS-EventLoopGroup-23-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-25 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:55974-HFileCleaner.large.0-1515828376824 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: M:0;asf911:35806-HFileCleaner.small.0-1515828368185 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-23-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:35806-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: M:0;asf911:35806-HFileCleaner.large.0-1515828368185 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-17 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:55974-HFileCleaner.small.0-1515828376824 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-23-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-26 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-18 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_593603700_1 at /127.0.0.1:40735 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.9@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-20-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-21-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-21-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-28 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-21-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:55974-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-23-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:55974-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-23-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-23 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-20-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.8@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_545386473_1 at /127.0.0.1:54843 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-20-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:35806-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-23-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to /127.0.0.1:47899 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.8 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-22-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-27 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.9 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-23-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-21-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-24 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-23-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-22-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2864 (was 2437) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=304 (was 200) - SystemLoadAverage LEAK? -, ProcessCount=270 (was 270), AvailableMemoryMB=10466 (was 11461) 2018-01-13 07:26:22,574 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2864 is superior to 1024 2018-01-13 07:26:22,606 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Thread=423, OpenFileDescriptor=2856, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=270, AvailableMemoryMB=10465 2018-01-13 07:26:22,606 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=2856 is superior to 1024 2018-01-13 07:26:22,628 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741886_1064{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:22,629 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741886_1064 size 7 2018-01-13 07:26:22,631 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:26:22,631 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:26:22,634 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:22,635 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:22,635 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:22,635 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:22,635 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:22,636 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:41428 2018-01-13 07:26:22,637 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,637 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,638 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,640 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,641 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:41428 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:22,657 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:414280x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:22,658 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:41428-0x160ee6894500049 connected 2018-01-13 07:26:22,782 DEBUG [main] zookeeper.ZKUtil(357): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,783 DEBUG [main] zookeeper.ZKUtil(357): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,783 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=41428 2018-01-13 07:26:22,785 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=41428 2018-01-13 07:26:22,786 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=41428 2018-01-13 07:26:22,786 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:22,788 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:22,788 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:22,789 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:22,789 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:22,789 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:22,790 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:35713 2018-01-13 07:26:22,790 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,791 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,792 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,793 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,795 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:35713 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:22,805 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:357130x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:22,806 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:35713-0x160ee689450004a connected 2018-01-13 07:26:22,815 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:26:22,819 DEBUG [main] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,819 DEBUG [main] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,820 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=35713 2018-01-13 07:26:22,820 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=35713 2018-01-13 07:26:22,821 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=35713 2018-01-13 07:26:22,822 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:22,831 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741873_1050, blk_1073741874_1051, blk_1073741875_1052, blk_1073741876_1053, blk_1073741877_1054, blk_1073741878_1055, blk_1073741879_1056, blk_1073741880_1057, blk_1073741881_1058, blk_1073741882_1059, blk_1073741883_1061, blk_1073741884_1062, blk_1073741885_1063] 2018-01-13 07:26:22,846 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:22,846 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:22,846 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:22,847 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:22,847 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:22,847 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:22,848 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:33436 2018-01-13 07:26:22,848 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,848 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,849 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,851 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,853 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:33436 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:22,863 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:334360x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:22,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:33436-0x160ee689450004b connected 2018-01-13 07:26:22,865 DEBUG [main] zookeeper.ZKUtil(357): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:22,865 DEBUG [main] zookeeper.ZKUtil(357): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:22,866 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=33436 2018-01-13 07:26:22,867 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=33436 2018-01-13 07:26:22,868 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=33436 2018-01-13 07:26:22,890 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:22,890 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:22,890 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:22,890 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:22,891 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:22,891 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:22,892 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:37733 2018-01-13 07:26:22,892 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,892 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:22,893 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,895 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:22,896 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:37733 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:23,013 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:377330x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:23,015 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:37733-0x160ee689450004c connected 2018-01-13 07:26:23,015 DEBUG [main] zookeeper.ZKUtil(357): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:23,016 DEBUG [main] zookeeper.ZKUtil(357): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:23,016 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=37733 2018-01-13 07:26:23,017 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=37733 2018-01-13 07:26:23,018 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=37733 2018-01-13 07:26:23,019 INFO [M:0;asf911:41428] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,41428,1515828382633 2018-01-13 07:26:23,019 INFO [M:1;asf911:35713] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,041 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(355): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,041 DEBUG [M:0;asf911:41428] zookeeper.ZKUtil(355): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,41428,1515828382633 2018-01-13 07:26:23,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:23,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:23,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:23,083 DEBUG [M:0;asf911:41428] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:26:23,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:23,085 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(355): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:23,087 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:23,089 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:23,089 DEBUG [M:0;asf911:41428] zookeeper.ZKUtil(355): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:23,091 INFO [M:1;asf911:35713] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,35713,1515828382787 from backup master directory 2018-01-13 07:26:23,091 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:23,091 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:23,091 INFO [M:0;asf911:41428] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,35713,1515828382787; waiting to become the next active master 2018-01-13 07:26:23,107 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,107 WARN [M:1;asf911:35713] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:23,107 INFO [M:1;asf911:35713] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,143 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741887_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:23,144 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741887_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:23,146 DEBUG [M:1;asf911:35713] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: aee7c76f-4316-44a8-84a0-cadfc8a570a3 2018-01-13 07:26:23,151 INFO [M:1;asf911:35713] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:26:23,152 INFO [M:1;asf911:35713] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:26:23,168 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741888_1066{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:23,168 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741888_1066{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:23,169 DEBUG [M:1;asf911:35713] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:23,184 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:23,185 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:23,185 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:23,185 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:23,189 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:23,189 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:23,190 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:23,190 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:23,194 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:23,194 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:23,194 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:23,194 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:23,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:23,198 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:23,198 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:23,199 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:23,202 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:23,203 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:23,203 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:23,203 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:23,205 DEBUG [M:1;asf911:35713] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:23,208 DEBUG [M:1;asf911:35713] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:23,212 DEBUG [M:1;asf911:35713] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:23,212 INFO [M:1;asf911:35713] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:26:23,212 DEBUG [M:1;asf911:35713] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:23,212 DEBUG [M:1;asf911:35713] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:23,212 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:23,213 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:23,213 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:23,213 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:23,213 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:23,213 INFO [M:1;asf911:35713] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:23,228 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741889_1067{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:23,229 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741889_1067 size 2318 2018-01-13 07:26:23,239 DEBUG [M:1;asf911:35713] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:23,264 INFO [M:1;asf911:35713] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:23,264 INFO [M:1;asf911:35713] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:23,265 DEBUG [M:1;asf911:35713] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:23,283 INFO [M:1;asf911:35713] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x61179e2c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:23,317 DEBUG [M:1;asf911:35713] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5db2e3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:23,340 INFO [M:1;asf911:35713] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:23,340 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:23,341 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:23,357 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:23,357 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:23,358 INFO [M:1;asf911:35713] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:23,361 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x1dc0f3bb no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:23,361 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x488d1093 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:23,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:23,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:23,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:23,382 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:23,390 INFO [M:1;asf911:35713] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,35713,1515828382787, sessionid=0x160ee689450004a, setting cluster-up flag (Was=false) 2018-01-13 07:26:23,473 INFO [M:1;asf911:35713] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:23,475 DEBUG [M:1;asf911:35713] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,540 INFO [M:1;asf911:35713] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:23,542 DEBUG [M:1;asf911:35713] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:23,543 INFO [M:1;asf911:35713] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:23,544 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:35713, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:23,544 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:35713, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:23,544 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:35713, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:23,544 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:35713, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:23,544 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:35713, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:23,545 DEBUG [M:1;asf911:35713] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:35713, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:23,547 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:23,547 INFO [M:1;asf911:35713] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:23,550 DEBUG [M:1;asf911:35713] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:26:23,551 INFO [M:1;asf911:35713] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:26:23,551 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 4msec 2018-01-13 07:26:23,552 DEBUG [M:1;asf911:35713] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:26:23,552 DEBUG [M:1;asf911:35713] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:26:23,552 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:26:23,554 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828413554 2018-01-13 07:26:23,554 INFO [M:1;asf911:35713] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:23,554 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:23,555 INFO [M:1;asf911:35713] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:23,555 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:23,555 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:23,555 INFO [M:1;asf911:35713] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:23,558 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:23,558 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:23,558 DEBUG [M:1;asf911:35713] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:23,558 DEBUG [M:1;asf911:35713] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:35713-HFileCleaner.large.0-1515828383558,5,main] 2018-01-13 07:26:23,559 DEBUG [M:1;asf911:35713] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:35713-HFileCleaner.small.0-1515828383559,5,main] 2018-01-13 07:26:23,559 INFO [M:1;asf911:35713] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828383041, completionTime=-1 2018-01-13 07:26:23,559 INFO [M:1;asf911:35713] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:23,565 DEBUG [M:1;asf911:35713-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:23,566 DEBUG [M:1;asf911:35713-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee689450004e connected 2018-01-13 07:26:23,627 INFO [RS:0;asf911:33436] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1c47425d to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:23,627 INFO [RS:1;asf911:37733] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x14c1a820 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:23,680 DEBUG [RS:0;asf911:33436] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1280bfa5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:23,681 DEBUG [RS:1;asf911:37733] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a0654c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:23,682 INFO [RS:0;asf911:33436] regionserver.HRegionServer(866): ClusterId : aee7c76f-4316-44a8-84a0-cadfc8a570a3 2018-01-13 07:26:23,682 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:23,698 DEBUG [RS:0;asf911:33436] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:23,698 INFO [RS:1;asf911:37733] regionserver.HRegionServer(866): ClusterId : aee7c76f-4316-44a8-84a0-cadfc8a570a3 2018-01-13 07:26:23,699 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:23,707 DEBUG [RS:1;asf911:37733] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:23,707 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:23,707 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:23,740 DEBUG [RS:0;asf911:33436] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:23,740 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:23,740 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:23,757 DEBUG [RS:1;asf911:37733] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:23,757 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:23,757 DEBUG [RS:0;asf911:33436] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16bf292d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:23,757 DEBUG [RS:0;asf911:33436] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:33436 2018-01-13 07:26:23,758 INFO [RS:0;asf911:33436] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:23,758 INFO [RS:0;asf911:33436] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:23,758 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:23,758 DEBUG [RS:1;asf911:37733] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e39c350, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:23,758 INFO [RS:0;asf911:33436] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,35713,1515828382787 with port=33436, startcode=1515828382845 2018-01-13 07:26:23,759 DEBUG [RS:1;asf911:37733] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:37733 2018-01-13 07:26:23,759 INFO [RS:1;asf911:37733] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:23,759 INFO [RS:1;asf911:37733] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:23,759 DEBUG [RS:0;asf911:33436] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35713 2018-01-13 07:26:23,760 INFO [RS:1;asf911:37733] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,35713,1515828382787 with port=37733, startcode=1515828382889 2018-01-13 07:26:23,760 DEBUG [RS:1;asf911:37733] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:35713 2018-01-13 07:26:23,761 INFO [RS-EventLoopGroup-25-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.10 (auth:SIMPLE) 2018-01-13 07:26:23,762 INFO [RS-EventLoopGroup-25-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 53112 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:23,764 INFO [RS-EventLoopGroup-25-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.11 (auth:SIMPLE) 2018-01-13 07:26:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35713] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,765 INFO [RS-EventLoopGroup-25-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 45710 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:23,765 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35713] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,765 DEBUG [RS:0;asf911:33436] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:23,766 DEBUG [RS:0;asf911:33436] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:23,766 DEBUG [RS:0;asf911:33436] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:23,766 DEBUG [RS:1;asf911:37733] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:23,766 DEBUG [RS:1;asf911:37733] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:23,766 DEBUG [RS:1;asf911:37733] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:23,790 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:23,810 INFO [M:1;asf911:35713] master.ServerManager(891): Finished wait on RegionServer count=2; waited=251ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:23,825 DEBUG [RS:0;asf911:33436] zookeeper.ZKUtil(355): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,825 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,825 WARN [RS:0;asf911:33436] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:23,825 DEBUG [RS:1;asf911:37733] zookeeper.ZKUtil(355): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,826 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,826 DEBUG [RS:0;asf911:33436] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,826 INFO [M:1;asf911:35713] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:23,826 WARN [RS:1;asf911:37733] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:23,828 DEBUG [RS:1;asf911:37733] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,837 INFO [RS:0;asf911:33436] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:23,837 INFO [RS:1;asf911:37733] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:23,839 DEBUG [RS:0;asf911:33436] zookeeper.ZKUtil(355): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,839 DEBUG [RS:1;asf911:37733] zookeeper.ZKUtil(355): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,839 DEBUG [RS:0;asf911:33436] zookeeper.ZKUtil(355): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,839 DEBUG [RS:1;asf911:37733] zookeeper.ZKUtil(355): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,840 INFO [RS:0;asf911:33436] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x33924ad1 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:23,840 INFO [RS:1;asf911:37733] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x3b14a1bc to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:23,891 DEBUG [RS:0;asf911:33436] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@345ba21d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:23,891 DEBUG [RS:0;asf911:33436] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:23,891 INFO [RS:0;asf911:33436] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:23,892 DEBUG [RS:1;asf911:37733] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0fe14f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:23,892 DEBUG [RS:1;asf911:37733] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:23,892 INFO [RS:1;asf911:37733] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:23,895 INFO [RS:0;asf911:33436] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:23,895 INFO [RS:0;asf911:33436] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:23,895 INFO [RS:0;asf911:33436] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:23,898 INFO [RS:1;asf911:37733] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:23,899 INFO [RS:1;asf911:37733] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:23,900 INFO [RS:1;asf911:37733] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:23,900 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:33436, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,901 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:33436, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:23,902 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:33436, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,903 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:33436, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,903 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:33436, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:23,903 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:33436, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:23,905 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:33436, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:23,905 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:33436, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,905 DEBUG [RS:0;asf911:33436] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:33436, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:23,905 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:37733, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,905 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:37733, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:23,905 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:37733, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:37733, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:37733, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:37733, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:37733, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:37733, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:23,906 DEBUG [RS:1;asf911:37733] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:37733, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:23,926 INFO [SplitLogWorker-asf911:37733] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,37733,1515828382889 starting 2018-01-13 07:26:23,926 INFO [SplitLogWorker-asf911:33436] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,33436,1515828382845 starting 2018-01-13 07:26:23,926 INFO [RS:0;asf911:33436] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:23,926 INFO [RS:1;asf911:37733] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:23,956 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,37733,1515828382889, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:37733, sessionid=0x160ee689450004c 2018-01-13 07:26:23,957 INFO [RS:0;asf911:33436] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,33436,1515828382845, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:33436, sessionid=0x160ee689450004b 2018-01-13 07:26:23,957 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:23,957 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:23,957 DEBUG [RS:0;asf911:33436] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,957 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,33436,1515828382845' 2018-01-13 07:26:23,957 DEBUG [RS:1;asf911:37733] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,957 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:23,957 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,37733,1515828382889' 2018-01-13 07:26:23,957 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:23,958 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:23,958 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:23,958 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:23,959 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:23,959 DEBUG [RS:0;asf911:33436] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:23,959 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,33436,1515828382845' 2018-01-13 07:26:23,959 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:23,958 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:23,959 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:23,959 DEBUG [RS:1;asf911:37733] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:23,959 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,37733,1515828382889' 2018-01-13 07:26:23,959 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:23,959 DEBUG [RS:0;asf911:33436] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:23,960 DEBUG [RS:0;asf911:33436] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:23,960 DEBUG [RS:1;asf911:37733] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:23,960 INFO [RS:0;asf911:33436] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:23,960 INFO [RS:0;asf911:33436] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:23,960 DEBUG [RS:1;asf911:37733] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:23,961 INFO [RS:1;asf911:37733] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:23,961 INFO [RS:1;asf911:37733] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:24,036 DEBUG [M:1;asf911:35713] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:24,038 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:24,038 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:24,039 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:24,041 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:26:24,100 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:24,101 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:24,101 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:26:24,101 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:26:24,138 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:24,138 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:26:24,289 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:26:24,289 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:26:24,297 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:24,965 INFO [RS:0;asf911:33436] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C33436%2C1515828382845, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33436,1515828382845, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:24,965 INFO [RS:1;asf911:37733] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C37733%2C1515828382889, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:24,979 DEBUG [RS-EventLoopGroup-27-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:24,980 DEBUG [RS-EventLoopGroup-27-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:24,980 DEBUG [RS-EventLoopGroup-27-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:24,981 DEBUG [RS-EventLoopGroup-27-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:24,984 INFO [RS:1;asf911:37733] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889/asf911.gq1.ygridcore.net%2C37733%2C1515828382889.1515828384966 2018-01-13 07:26:24,984 INFO [RS:0;asf911:33436] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33436,1515828382845/asf911.gq1.ygridcore.net%2C33436%2C1515828382845.1515828384966 2018-01-13 07:26:24,984 DEBUG [RS:1;asf911:37733] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:24,984 DEBUG [RS:0;asf911:33436] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK]] 2018-01-13 07:26:24,987 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:24,998 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:26:25,007 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,159 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,37733,1515828382889 version=3145728 2018-01-13 07:26:25,160 DEBUG [RSProcedureDispatcher-pool52-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,161 DEBUG [RSProcedureDispatcher-pool52-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:25,163 INFO [RS-EventLoopGroup-27-13] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:25,163 INFO [RS-EventLoopGroup-27-13] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 42211 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:25,165 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=37733] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:25,166 INFO [RS_OPEN_META-asf911:37733-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:25,170 INFO [RS_OPEN_META-asf911:37733-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C37733%2C1515828382889.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:25,176 DEBUG [RS-EventLoopGroup-27-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:25,176 DEBUG [RS-EventLoopGroup-27-15] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:25,181 INFO [RS_OPEN_META-asf911:37733-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889/asf911.gq1.ygridcore.net%2C37733%2C1515828382889.meta.1515828385170.meta 2018-01-13 07:26:25,181 DEBUG [RS_OPEN_META-asf911:37733-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:25,181 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:25,181 DEBUG [RS_OPEN_META-asf911:37733-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:25,182 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:25,182 INFO [RS_OPEN_META-asf911:37733-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:25,182 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:25,182 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:25,187 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:25,187 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:25,187 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:25,188 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:25,188 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:25,191 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:25,191 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:25,191 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:25,191 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:25,192 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:25,194 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:25,195 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:25,195 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:25,195 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:25,195 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:25,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:25,198 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:25,198 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:25,198 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:25,198 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:25,201 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:25,201 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:25,201 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:25,201 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:25,202 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:25,205 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:25,207 DEBUG [RS_OPEN_META-asf911:37733-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:25,212 DEBUG [RS_OPEN_META-asf911:37733-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:25,213 INFO [RS_OPEN_META-asf911:37733-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:26:25,215 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:25,220 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=35713] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,221 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828385220}]},"ts":1515828385220} 2018-01-13 07:26:25,221 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:25,223 DEBUG [RS_OPEN_META-asf911:37733-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,226 DEBUG [hconnection-0x66f1b524-shared-pool55-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:25,228 INFO [RS-EventLoopGroup-27-19] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:25,229 INFO [RS-EventLoopGroup-27-19] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 42214 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:25,232 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:25,233 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,248 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:25,292 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:26:25,333 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:25,333 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.1470sec 2018-01-13 07:26:25,375 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.5050sec 2018-01-13 07:26:25,376 INFO [M:1;asf911:35713] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:25,387 INFO [M:1;asf911:35713] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33436,1515828382845 belongs to an existing region server 2018-01-13 07:26:25,388 INFO [M:1;asf911:35713] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,37733,1515828382889 belongs to an existing region server 2018-01-13 07:26:25,388 INFO [M:1;asf911:35713] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:25,390 INFO [M:1;asf911:35713] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:25,390 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828390390 2018-01-13 07:26:25,390 INFO [M:1;asf911:35713] assignment.AssignmentManager(1219): Joined the cluster in 2msec, failover=true 2018-01-13 07:26:25,395 INFO [M:1;asf911:35713] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:26:25,395 INFO [M:1;asf911:35713] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:26:25,533 DEBUG [M:1;asf911:35713] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:25,535 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:25,584 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:25,606 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741894_1072{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 476 2018-01-13 07:26:25,606 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741894_1072 size 476 2018-01-13 07:26:25,832 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741873_1050, blk_1073741874_1051, blk_1073741875_1052, blk_1073741876_1053, blk_1073741877_1054, blk_1073741878_1055, blk_1073741879_1056, blk_1073741880_1057, blk_1073741881_1058, blk_1073741882_1059, blk_1073741883_1061, blk_1073741884_1062, blk_1073741885_1063] 2018-01-13 07:26:26,010 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:26,012 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:26:26,028 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741895_1073{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 42 2018-01-13 07:26:26,028 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741895_1073 size 42 2018-01-13 07:26:26,258 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(649): Not transferring queue since we are shutting down 2018-01-13 07:26:26,429 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:26,430 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.: disabling compactions & flushes 2018-01-13 07:26:26,430 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:26,430 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:26,531 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:26,533 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:26,538 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:26,606 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:26,606 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828386606}]},"ts":1515828386606} 2018-01-13 07:26:26,611 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:26:26,611 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:26,619 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889}] 2018-01-13 07:26:26,678 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889 hbase:namespace hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:26,681 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889; rit=OFFLINE, location=asf911.gq1.ygridcore.net,37733,1515828382889; forceNewPlan=false, retain=false 2018-01-13 07:26:26,834 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:26,837 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:26,838 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1db5f065 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:26,857 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13721da3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:26,859 DEBUG [MultiHConnection-shared--pool56-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:26,861 INFO [RS-EventLoopGroup-27-21] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:26,861 INFO [RS-EventLoopGroup-27-21] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 42231 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:26,864 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889; rit=OPENING, location=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:27,015 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,37733,1515828382889 version=3145728 2018-01-13 07:26:27,018 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=37733] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:27,024 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] regionserver.HRegion(6949): Opening region: {ENCODED => 46f0ed55c1ffea8aab1f6e40ed1ae924, NAME => 'hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:27,024 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 46f0ed55c1ffea8aab1f6e40ed1ae924 2018-01-13 07:26:27,024 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:27,029 DEBUG [StoreOpener-46f0ed55c1ffea8aab1f6e40ed1ae924-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/46f0ed55c1ffea8aab1f6e40ed1ae924/info 2018-01-13 07:26:27,029 DEBUG [StoreOpener-46f0ed55c1ffea8aab1f6e40ed1ae924-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/46f0ed55c1ffea8aab1f6e40ed1ae924/info 2018-01-13 07:26:27,030 INFO [StoreOpener-46f0ed55c1ffea8aab1f6e40ed1ae924-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:27,030 INFO [StoreOpener-46f0ed55c1ffea8aab1f6e40ed1ae924-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:27,030 INFO [StoreOpener-46f0ed55c1ffea8aab1f6e40ed1ae924-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:27,032 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/46f0ed55c1ffea8aab1f6e40ed1ae924 2018-01-13 07:26:27,039 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/46f0ed55c1ffea8aab1f6e40ed1ae924/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:27,039 INFO [RS_OPEN_PRIORITY_REGION-asf911:37733-0] regionserver.HRegion(960): Onlined 46f0ed55c1ffea8aab1f6e40ed1ae924; next sequenceid=2 2018-01-13 07:26:27,046 INFO [PostOpenDeployTasks:46f0ed55c1ffea8aab1f6e40ed1ae924] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:27,049 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=35713] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889; rit=OPENING, location=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:27,050 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:27,050 DEBUG [PostOpenDeployTasks:46f0ed55c1ffea8aab1f6e40ed1ae924] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:27,053 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:37733-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. on asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:27,107 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:27,111 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:27,162 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:26:27,162 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:27,162 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=46f0ed55c1ffea8aab1f6e40ed1ae924, target=asf911.gq1.ygridcore.net,37733,1515828382889 in 441msec 2018-01-13 07:26:27,162 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828387162}]},"ts":1515828387162} 2018-01-13 07:26:27,168 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:26:27,239 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:27,258 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:26:27,280 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:27,293 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.8450sec 2018-01-13 07:26:27,335 DEBUG [M:1;asf911:35713] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,336 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,385 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,426 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,480 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,490 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:27,547 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:27,597 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 260msec 2018-01-13 07:26:27,747 DEBUG [M:1;asf911:35713] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:27,749 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:27,797 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:27,838 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:27,888 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:27,898 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:27,960 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:28,001 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 360msec 2018-01-13 07:26:28,023 DEBUG [M:1;asf911:35713] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:28,032 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:28,040 DEBUG [M:1;asf911:35713] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:28,048 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:28,048 INFO [M:1;asf911:35713] master.HMaster(928): Master has completed initialization 4.941sec 2018-01-13 07:26:28,049 INFO [M:1;asf911:35713] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:28,050 INFO [M:1;asf911:35713] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:28,051 INFO [M:1;asf911:35713] regionserver.HRegionServer(866): ClusterId : aee7c76f-4316-44a8-84a0-cadfc8a570a3 2018-01-13 07:26:28,130 DEBUG [hconnection-0x51a18b21-shared-pool12-t6] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:28,132 DEBUG [Default-IPC-NioEventLoopGroup-7-19] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:51604 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:51604 2018-01-13 07:26:28,237 DEBUG [hconnection-0x51a18b21-shared-pool12-t6] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:28,240 INFO [RS-EventLoopGroup-27-22] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:28,240 INFO [RS-EventLoopGroup-27-22] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 42242 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:28,244 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:28,244 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:28,255 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:28,257 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE-0x160ee6894500054 connected 2018-01-13 07:26:28,257 DEBUG [main] zookeeper.ZKUtil(446): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE-0x160ee6894500054, quorum=localhost:59094, baseZNode=/hbase Unable to list children of znode /wrongNode because node does not exist (not an error) 2018-01-13 07:26:28,257 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,33436,1515828382845: killAll ***** 2018-01-13 07:26:28,257 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:28,263 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 433297440 }, "NonHeapMemoryUsage" : { "committed" : 130613248, "init" : 2555904, "max" : -1, "used" : 128676528 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 44161, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 201, "TotalCallTime_min" : 0, "TotalCallTime_max" : 5, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 5, "TotalCallTime_99th_percentile" : 5, "TotalCallTime_99.9th_percentile" : 5, "TotalCallTime_TimeRangeCount_0-1" : 10, "TotalCallTime_TimeRangeCount_1-3" : 6, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 182, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 201, "ResponseSize_min" : 0, "ResponseSize_max" : 72, "ResponseSize_mean" : 27, "ResponseSize_25th_percentile" : 7, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 58, "ResponseSize_95th_percentile" : 70, "ResponseSize_98th_percentile" : 71, "ResponseSize_99th_percentile" : 71, "ResponseSize_99.9th_percentile" : 71, "ResponseSize_SizeRangeCount_0-10" : 10, "ResponseSize_SizeRangeCount_10-100" : 9, "ResponseSize_SizeRangeCount_100000000-inf" : 182, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 201, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 5, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 5, "ProcessCallTime_98th_percentile" : 5, "ProcessCallTime_99th_percentile" : 5, "ProcessCallTime_99.9th_percentile" : 5, "ProcessCallTime_TimeRangeCount_0-1" : 10, "ProcessCallTime_TimeRangeCount_1-3" : 6, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 34, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 201, "RequestSize_min" : 78, "RequestSize_max" : 737, "RequestSize_mean" : 164, "RequestSize_25th_percentile" : 84, "RequestSize_median" : 86, "RequestSize_75th_percentile" : 131, "RequestSize_90th_percentile" : 516, "RequestSize_95th_percentile" : 735, "RequestSize_98th_percentile" : 736, "RequestSize_99th_percentile" : 736, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 13, "RequestSize_SizeRangeCount_100-1000" : 6, "RequestSize_SizeRangeCount_100000000-inf" : 182, "sentBytes" : 19324, "QueueCallTime_num_ops" : 201, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 19, "QueueCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:28,269 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35713] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,33436,1515828382845 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,33436,1515828382845: killAll ***** 2018-01-13 07:26:28,269 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,33436,1515828382845' ***** 2018-01-13 07:26:28,270 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:28,270 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,37733,1515828382889: killAll ***** 2018-01-13 07:26:28,270 INFO [RS:0;asf911:33436] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:28,270 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:28,270 INFO [SplitLogWorker-asf911:33436] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:28,270 INFO [SplitLogWorker-asf911:33436] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,33436,1515828382845 exiting 2018-01-13 07:26:28,270 INFO [RS:0;asf911:33436] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:28,271 INFO [RS:0;asf911:33436] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:28,272 INFO [RS:0;asf911:33436] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:28,272 INFO [RS:0;asf911:33436] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:28,273 DEBUG [RS:0;asf911:33436] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:28,272 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:28,271 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:28,273 INFO [RS:0;asf911:33436] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x1c47425d to localhost:59094 2018-01-13 07:26:28,273 DEBUG [RS:0;asf911:33436] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,273 INFO [RS:0;asf911:33436] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,33436,1515828382845; all regions closed. 2018-01-13 07:26:28,273 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 433322040 }, "NonHeapMemoryUsage" : { "committed" : 130613248, "init" : 2555904, "max" : -1, "used" : 128673072 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 44161, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 201, "TotalCallTime_min" : 0, "TotalCallTime_max" : 5, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 5, "TotalCallTime_99th_percentile" : 5, "TotalCallTime_99.9th_percentile" : 5, "TotalCallTime_TimeRangeCount_0-1" : 10, "TotalCallTime_TimeRangeCount_1-3" : 6, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 182, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 201, "ResponseSize_min" : 0, "ResponseSize_max" : 72, "ResponseSize_mean" : 27, "ResponseSize_25th_percentile" : 7, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 58, "ResponseSize_95th_percentile" : 70, "ResponseSize_98th_percentile" : 71, "ResponseSize_99th_percentile" : 71, "ResponseSize_99.9th_percentile" : 71, "ResponseSize_SizeRangeCount_0-10" : 10, "ResponseSize_SizeRangeCount_10-100" : 9, "ResponseSize_SizeRangeCount_100000000-inf" : 182, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 201, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 5, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 5, "ProcessCallTime_98th_percentile" : 5, "ProcessCallTime_99th_percentile" : 5, "ProcessCallTime_99.9th_percentile" : 5, "ProcessCallTime_TimeRangeCount_0-1" : 10, "ProcessCallTime_TimeRangeCount_1-3" : 6, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 34, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 201, "RequestSize_min" : 78, "RequestSize_max" : 737, "RequestSize_mean" : 164, "RequestSize_25th_percentile" : 84, "RequestSize_median" : 86, "RequestSize_75th_percentile" : 131, "RequestSize_90th_percentile" : 516, "RequestSize_95th_percentile" : 735, "RequestSize_98th_percentile" : 736, "RequestSize_99th_percentile" : 736, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 13, "RequestSize_SizeRangeCount_100-1000" : 6, "RequestSize_SizeRangeCount_100000000-inf" : 182, "sentBytes" : 19324, "QueueCallTime_num_ops" : 201, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 19, "QueueCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:28,276 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35713] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,37733,1515828382889 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,37733,1515828382889: killAll ***** 2018-01-13 07:26:28,276 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,37733,1515828382889' ***** 2018-01-13 07:26:28,277 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:28,277 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:28,279 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,41428,1515828382633' ***** 2018-01-13 07:26:28,280 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:28,277 INFO [RS:1;asf911:37733] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:28,280 INFO [RS:1;asf911:37733] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:28,281 INFO [SplitLogWorker-asf911:37733] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:28,288 INFO [RS:1;asf911:37733] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:28,288 INFO [RS:1;asf911:37733] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:28,286 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:28,288 INFO [SplitLogWorker-asf911:37733] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,37733,1515828382889 exiting 2018-01-13 07:26:28,288 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:28,289 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:28,291 DEBUG [RS:1;asf911:37733] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:28,289 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:28,291 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:28,292 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,35713,1515828382787' ***** 2018-01-13 07:26:28,292 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:28,289 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741891_1069{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,291 INFO [RS:1;asf911:37733] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x14c1a820 to localhost:59094 2018-01-13 07:26:28,291 INFO [M:0;asf911:41428] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x27f6781c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:28,291 DEBUG [RS_CLOSE_REGION-asf911:37733-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:28,292 DEBUG [RS:1;asf911:37733] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,292 DEBUG [M:1;asf911:35713] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72e08661, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:28,292 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741891_1069{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,294 INFO [M:1;asf911:35713] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:28,294 DEBUG [M:1;asf911:35713] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:28,294 INFO [RS:1;asf911:37733] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:28,295 INFO [RS:1;asf911:37733] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:28,295 INFO [RS:1;asf911:37733] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:28,294 DEBUG [RS_CLOSE_REGION-asf911:37733-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.: disabling compactions & flushes 2018-01-13 07:26:28,295 DEBUG [RS_CLOSE_REGION-asf911:37733-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:28,295 INFO [M:1;asf911:35713] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x61179e2c to localhost:59094 2018-01-13 07:26:28,295 DEBUG [M:1;asf911:35713] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,296 INFO [M:1;asf911:35713] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,35713,1515828382787; all regions closed. 2018-01-13 07:26:28,296 DEBUG [M:1;asf911:35713] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,296 INFO [StoreCloserThread-hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:28,296 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:26:28,296 DEBUG [RS:1;asf911:37733] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740, 46f0ed55c1ffea8aab1f6e40ed1ae924=hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924.} 2018-01-13 07:26:28,296 ERROR [RS_CLOSE_REGION-asf911:37733-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:26:28,296 DEBUG [RS_CLOSE_META-asf911:37733-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:28,298 INFO [RS_CLOSE_REGION-asf911:37733-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:28,296 INFO [M:1;asf911:35713] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,35713,1515828382787 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:35713 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35713,1515828382787-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35713,1515828382787-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35713,1515828382787-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35713,1515828382787-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,35713,1515828382787-MobCompactionChore Period: 604800 Unit: SECONDS]] on shutdown 2018-01-13 07:26:28,301 DEBUG [RS_CLOSE_REGION-asf911:37733-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828385395.46f0ed55c1ffea8aab1f6e40ed1ae924. 2018-01-13 07:26:28,301 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:28,300 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:28,300 DEBUG [RS_CLOSE_META-asf911:37733-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:28,299 DEBUG [RS:0;asf911:33436] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,301 DEBUG [RS_CLOSE_META-asf911:37733-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:28,301 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:26:28,301 INFO [M:1;asf911:35713] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:28,303 INFO [M:1;asf911:35713] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:28,303 DEBUG [M:1;asf911:35713] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:28,301 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:28,301 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:28,303 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:28,303 INFO [RS:0;asf911:33436] regionserver.Leases(147): RS:0;asf911:33436 closing leases 2018-01-13 07:26:28,304 INFO [RS:0;asf911:33436] regionserver.Leases(150): RS:0;asf911:33436 closed leases 2018-01-13 07:26:28,304 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:28,304 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:28,304 INFO [RS:0;asf911:33436] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,33436,1515828382845 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,33436,1515828382845 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:28,304 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:28,305 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:28,305 INFO [RS:0;asf911:33436] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:28,305 INFO [RS:0;asf911:33436] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:28,305 INFO [RS:0;asf911:33436] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:28,305 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:28,306 INFO [RS:0;asf911:33436] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:28,306 ERROR [RS_CLOSE_META-asf911:37733-0] regionserver.HRegion(1620): Memstore size is 1330 2018-01-13 07:26:28,307 DEBUG [RS_CLOSE_META-asf911:37733-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:28,309 INFO [RS_CLOSE_META-asf911:37733-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:28,309 DEBUG [RS_CLOSE_META-asf911:37733-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:28,312 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:28,312 DEBUG [M:1;asf911:35713] zookeeper.ZKUtil(614): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:28,312 DEBUG [main] zookeeper.ZKUtil(614): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:28,313 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:28,313 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,35713,1515828382787 2018-01-13 07:26:28,312 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:28,312 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:28,312 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:28,312 WARN [M:1;asf911:35713] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:28,314 INFO [M:1;asf911:35713] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:28,314 INFO [M:1;asf911:35713] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x1db5f065 to localhost:59094 2018-01-13 07:26:28,314 DEBUG [M:1;asf911:35713] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,314 INFO [M:1;asf911:35713] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:28,316 INFO [M:1;asf911:35713] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:28,316 INFO [M:1;asf911:35713] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:28,317 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,319 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,321 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,326 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:28,316 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,326 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:28,326 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:28,326 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,33436,1515828382845 2018-01-13 07:26:28,326 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,321 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,321 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,321 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,321 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,320 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,318 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:28,328 DEBUG [M:0;asf911:41428] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29318292, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:28,328 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:28,328 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,33436,1515828382845] 2018-01-13 07:26:28,332 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:28,332 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:28,332 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:28,334 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; asf911.gq1.ygridcore.net,33436,1515828382845 expired; onlineServers=1 2018-01-13 07:26:28,335 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:28,336 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:28,343 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741890_1068{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:28,344 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741890_1068{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:26:28,345 INFO [M:1;asf911:35713] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,35713,1515828382787_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:28,345 INFO [M:1;asf911:35713] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:28,345 INFO [M:1;asf911:35713] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:28,348 INFO [RS:0;asf911:33436] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,33436,1515828382845; zookeeper connection closed. 2018-01-13 07:26:28,348 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:28,348 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:35713-0x160ee689450004a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:28,348 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:28,348 WARN [main-EventThread] zookeeper.ZKUtil(361): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Unable to set watcher on znode /hbase/running org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:28,348 INFO [RS:0;asf911:33436] regionserver.HRegionServer(1157): RS:0;asf911:33436 exiting 2018-01-13 07:26:28,349 ERROR [main-EventThread] zookeeper.ZKWatcher(578): regionserver:33436-0x160ee689450004b, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:28,350 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6cb7166f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6cb7166f 2018-01-13 07:26:28,350 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,33436,1515828382845: Unexpected exception handling nodeDeleted event ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/running at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:28,350 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:28,353 INFO [main-EventThread] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 444391848 }, "NonHeapMemoryUsage" : { "committed" : 130678784, "init" : 2555904, "max" : -1, "used" : 128776640 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 44161, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 201, "TotalCallTime_min" : 0, "TotalCallTime_max" : 5, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 4, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 5, "TotalCallTime_99th_percentile" : 5, "TotalCallTime_99.9th_percentile" : 5, "TotalCallTime_TimeRangeCount_0-1" : 10, "TotalCallTime_TimeRangeCount_1-3" : 6, "TotalCallTime_TimeRangeCount_3-10" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 182, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 201, "ResponseSize_min" : 0, "ResponseSize_max" : 72, "ResponseSize_mean" : 27, "ResponseSize_25th_percentile" : 7, "ResponseSize_median" : 10, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 58, "ResponseSize_95th_percentile" : 70, "ResponseSize_98th_percentile" : 71, "ResponseSize_99th_percentile" : 71, "ResponseSize_99.9th_percentile" : 71, "ResponseSize_SizeRangeCount_0-10" : 10, "ResponseSize_SizeRangeCount_10-100" : 9, "ResponseSize_SizeRangeCount_100000000-inf" : 182, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 201, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 5, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 4, "ProcessCallTime_95th_percentile" : 5, "ProcessCallTime_98th_percentile" : 5, "ProcessCallTime_99th_percentile" : 5, "ProcessCallTime_99.9th_percentile" : 5, "ProcessCallTime_TimeRangeCount_0-1" : 10, "ProcessCallTime_TimeRangeCount_1-3" : 6, "ProcessCallTime_TimeRangeCount_3-10" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 34, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 201, "RequestSize_min" : 78, "RequestSize_max" : 737, "RequestSize_mean" : 164, "RequestSize_25th_percentile" : 84, "RequestSize_median" : 86, "RequestSize_75th_percentile" : 131, "RequestSize_90th_percentile" : 516, "RequestSize_95th_percentile" : 735, "RequestSize_98th_percentile" : 736, "RequestSize_99th_percentile" : 736, "RequestSize_99.9th_percentile" : 736, "RequestSize_SizeRangeCount_10-100" : 13, "RequestSize_SizeRangeCount_100-1000" : 6, "RequestSize_SizeRangeCount_100000000-inf" : 182, "sentBytes" : 19324, "QueueCallTime_num_ops" : 201, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 19, "QueueCallTime_TimeRangeCount_600000-inf" : 182, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:28,357 DEBUG [M:1;asf911:35713] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,35713,1515828382787 already deleted, retry=false 2018-01-13 07:26:28,365 INFO [M:1;asf911:35713] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,35713,1515828382787; zookeeper connection closed. 2018-01-13 07:26:28,366 INFO [M:1;asf911:35713] regionserver.HRegionServer(1157): M:1;asf911:35713 exiting 2018-01-13 07:26:28,497 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,37733,1515828382889; all regions closed. 2018-01-13 07:26:28,502 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741893_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,503 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741893_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,509 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741892_1070{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,509 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741892_1070{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:28,510 DEBUG [RS:1;asf911:37733] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:28,510 INFO [RS:1;asf911:37733] regionserver.Leases(147): RS:1;asf911:37733 closing leases 2018-01-13 07:26:28,510 INFO [RS:1;asf911:37733] regionserver.Leases(150): RS:1;asf911:37733 closed leases 2018-01-13 07:26:28,511 INFO [RS:1;asf911:37733] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,37733,1515828382889 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,37733,1515828382889 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:28,511 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:28,511 INFO [RS:1;asf911:37733] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:28,523 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:37733-0x160ee689450004c, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,37733,1515828382889 2018-01-13 07:26:28,532 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,37733,1515828382889; zookeeper connection closed. 2018-01-13 07:26:28,533 INFO [RS:1;asf911:37733] regionserver.HRegionServer(1157): RS:1;asf911:37733 exiting 2018-01-13 07:26:28,533 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@37cadffd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@37cadffd 2018-01-13 07:26:28,533 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:26:28,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:28,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:28,908 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:28,908 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:29,335 INFO [M:0;asf911:41428] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:29,337 INFO [M:0;asf911:41428] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,41428,1515828382633 2018-01-13 07:26:29,337 DEBUG [M:0;asf911:41428] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:29,337 INFO [M:0;asf911:41428] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x27f6781c to localhost:59094 2018-01-13 07:26:29,337 DEBUG [M:0;asf911:41428] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:29,337 INFO [M:0;asf911:41428] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,41428,1515828382633; all regions closed. 2018-01-13 07:26:29,338 INFO [M:0;asf911:41428] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,41428,1515828382633 had [] on shutdown 2018-01-13 07:26:29,338 DEBUG [M:0;asf911:41428] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:29,348 DEBUG [M:0;asf911:41428] zookeeper.ZKUtil(614): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:29,350 WARN [M:0;asf911:41428] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:29,382 DEBUG [M:0;asf911:41428] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,41428,1515828382633 already deleted, retry=false 2018-01-13 07:26:29,407 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:41428-0x160ee6894500049, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,41428,1515828382633 2018-01-13 07:26:29,407 INFO [M:0;asf911:41428] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,41428,1515828382633; zookeeper connection closed. 2018-01-13 07:26:29,408 INFO [M:0;asf911:41428] regionserver.HRegionServer(1157): M:0;asf911:41428 exiting 2018-01-13 07:26:29,416 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,418 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741890_1068 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,418 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741891_1069 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:29,418 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741892_1070 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741893_1071 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741889_1067 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741888_1066 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741894_1072 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741895_1073 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:29,419 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741887_1065 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,420 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741886_1064 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:29,420 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:29,432 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:29,435 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500056 connected 2018-01-13 07:26:29,436 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:26:29,436 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:26:29,437 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:26:29,437 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:26:29,438 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:26:29,438 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:26:29,439 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:26:29,439 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:26:29,440 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:26:29,440 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:26:29,441 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:29,441 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:26:29,442 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:26:29,445 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:26:29,445 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:26:29,446 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:26:29,447 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:26:29,447 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:26:29,448 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:26:29,449 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:26:29,450 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:26:29,463 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:29,464 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:26:29,471 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:26:29,471 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:29,471 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:29,471 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:29,471 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500056, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:26:29,510 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Thread=464 (was 423) Potentially hanging thread: RS-EventLoopGroup-27-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:35713-HFileCleaner.large.0-1515828383558 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-27-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.11 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-20 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-24-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:35713-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-27-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.11@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-25-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.10@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:35713-HFileCleaner.small.0-1515828383559 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-25-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:35713-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-27-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1472951736_1 at /127.0.0.1:40830 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-25-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-19 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.10 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-27-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-26-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-27-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=3260 (was 2856) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=328 (was 304) - SystemLoadAverage LEAK? -, ProcessCount=270 (was 270), AvailableMemoryMB=10349 (was 10465) 2018-01-13 07:26:29,511 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=3260 is superior to 1024 2018-01-13 07:26:29,546 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=463, OpenFileDescriptor=3252, MaxFileDescriptor=60000, SystemLoadAverage=328, ProcessCount=270, AvailableMemoryMB=10348 2018-01-13 07:26:29,547 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=3252 is superior to 1024 2018-01-13 07:26:29,562 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741896_1074{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:29,562 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741896_1074{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:29,564 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:26:29,565 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:26:29,568 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:29,569 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:29,569 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:29,569 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:29,569 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:29,571 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:42935 2018-01-13 07:26:29,571 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,571 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,572 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,574 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,576 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:42935 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:29,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:429350x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:29,583 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:42935-0x160ee6894500057 connected 2018-01-13 07:26:29,674 DEBUG [main] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:29,675 DEBUG [main] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:29,675 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=42935 2018-01-13 07:26:29,677 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=42935 2018-01-13 07:26:29,677 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=42935 2018-01-13 07:26:29,678 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:29,680 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:29,680 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:29,680 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:29,681 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:29,681 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:29,682 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:58949 2018-01-13 07:26:29,682 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,683 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,684 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,685 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,687 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:58949 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:29,699 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:589490x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:29,701 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:58949-0x160ee6894500058 connected 2018-01-13 07:26:29,707 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:26:29,710 DEBUG [main] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:29,711 DEBUG [main] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:29,711 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=58949 2018-01-13 07:26:29,712 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=58949 2018-01-13 07:26:29,713 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=58949 2018-01-13 07:26:29,714 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:29,734 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:29,735 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:29,735 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:29,735 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:29,735 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:29,736 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:29,737 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:33483 2018-01-13 07:26:29,737 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,737 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,738 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,740 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,741 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:33483 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:29,749 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:334830x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:29,750 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:33483-0x160ee6894500059 connected 2018-01-13 07:26:29,750 DEBUG [main] zookeeper.ZKUtil(357): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:29,751 DEBUG [main] zookeeper.ZKUtil(357): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:29,751 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=33483 2018-01-13 07:26:29,752 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=33483 2018-01-13 07:26:29,753 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=33483 2018-01-13 07:26:29,779 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:29,780 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:29,780 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:29,780 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:29,780 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:29,780 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:29,781 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:47183 2018-01-13 07:26:29,782 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,782 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:29,783 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,784 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:29,785 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:47183 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:29,796 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:471830x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:29,798 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:47183-0x160ee689450005a connected 2018-01-13 07:26:29,798 DEBUG [main] zookeeper.ZKUtil(357): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:29,799 DEBUG [main] zookeeper.ZKUtil(357): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:29,799 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=47183 2018-01-13 07:26:29,800 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=47183 2018-01-13 07:26:29,802 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=47183 2018-01-13 07:26:29,803 INFO [M:0;asf911:42935] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:29,803 INFO [M:1;asf911:58949] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:29,813 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(355): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:29,813 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:29,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:29,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:29,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:29,823 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:29,832 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:26:29,834 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(355): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:29,834 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:29,835 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:29,836 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:29,836 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:29,836 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:29,836 INFO [M:0;asf911:42935] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,58949,1515828389678; waiting to become the next active master 2018-01-13 07:26:29,836 INFO [M:1;asf911:58949] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,58949,1515828389678 from backup master directory 2018-01-13 07:26:29,846 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:29,846 WARN [M:1;asf911:58949] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:29,846 INFO [M:1;asf911:58949] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:29,874 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741897_1075{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 42 2018-01-13 07:26:29,875 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741897_1075 size 42 2018-01-13 07:26:30,279 DEBUG [M:1;asf911:58949] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 58f2da67-30c0-4081-84ee-cf0400bad60b 2018-01-13 07:26:30,287 INFO [M:1;asf911:58949] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:26:30,288 INFO [M:1;asf911:58949] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:26:30,321 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741898_1076{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 32 2018-01-13 07:26:30,321 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741898_1076 size 32 2018-01-13 07:26:30,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(156): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:30,491 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x377d8602 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:30,492 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x2e503175 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:30,723 DEBUG [M:1;asf911:58949] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:30,731 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:30,731 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:30,732 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:30,732 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:30,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:30,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:30,741 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:30,741 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:30,744 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:30,745 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:30,745 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:30,745 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:30,749 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:30,749 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:30,749 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:30,750 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:30,752 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:30,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:30,753 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:30,753 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:30,755 DEBUG [M:1;asf911:58949] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:30,757 DEBUG [M:1;asf911:58949] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:30,760 DEBUG [M:1;asf911:58949] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:30,760 INFO [M:1;asf911:58949] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:26:30,760 DEBUG [M:1;asf911:58949] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:30,760 DEBUG [M:1;asf911:58949] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:30,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:30,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:30,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:30,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:30,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:30,761 INFO [M:1;asf911:58949] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:30,772 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741899_1077{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:30,772 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741899_1077{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:30,775 DEBUG [M:1;asf911:58949] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:30,789 INFO [M:1;asf911:58949] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:30,790 INFO [M:1;asf911:58949] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:30,790 DEBUG [M:1;asf911:58949] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:30,806 INFO [M:1;asf911:58949] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x23e1792e to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:30,816 DEBUG [M:1;asf911:58949] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f056f6f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:30,844 INFO [M:1;asf911:58949] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:30,844 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:30,845 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:30,857 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:30,858 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:30,858 INFO [M:1;asf911:58949] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:30,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:30,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:30,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:30,865 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:30,865 INFO [M:1;asf911:58949] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,58949,1515828389678, sessionid=0x160ee6894500058, setting cluster-up flag (Was=false) 2018-01-13 07:26:30,915 INFO [M:1;asf911:58949] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:30,916 DEBUG [M:1;asf911:58949] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:30,957 INFO [M:1;asf911:58949] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:30,958 DEBUG [M:1;asf911:58949] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:30,959 INFO [M:1;asf911:58949] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:58949, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:58949, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:58949, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:58949, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:58949, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:30,960 DEBUG [M:1;asf911:58949] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:58949, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:30,962 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:30,963 INFO [M:1;asf911:58949] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:30,965 DEBUG [M:1;asf911:58949] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:26:30,966 INFO [M:1;asf911:58949] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:26:30,966 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 3msec 2018-01-13 07:26:30,966 DEBUG [M:1;asf911:58949] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:26:30,966 DEBUG [M:1;asf911:58949] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:26:30,967 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 1msec 2018-01-13 07:26:30,969 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828420969 2018-01-13 07:26:30,969 INFO [M:1;asf911:58949] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:30,969 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:30,970 INFO [M:1;asf911:58949] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:30,971 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:30,971 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:30,971 INFO [M:1;asf911:58949] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:30,973 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:30,981 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:30,981 DEBUG [M:1;asf911:58949] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:30,981 DEBUG [M:1;asf911:58949] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:58949-HFileCleaner.large.0-1515828390981,5,main] 2018-01-13 07:26:30,981 DEBUG [M:1;asf911:58949-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:30,987 DEBUG [M:1;asf911:58949-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee689450005c connected 2018-01-13 07:26:30,987 DEBUG [M:1;asf911:58949] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:58949-HFileCleaner.small.0-1515828390982,5,main] 2018-01-13 07:26:30,987 INFO [M:1;asf911:58949] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828389813, completionTime=-1 2018-01-13 07:26:30,987 INFO [M:1;asf911:58949] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:31,014 INFO [RS:0;asf911:33483] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x47f84e1b to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:31,014 INFO [RS:1;asf911:47183] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5638f7e5 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:31,032 DEBUG [RS:1;asf911:47183] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58ff1352, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:31,033 DEBUG [RS:0;asf911:33483] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@493fb894, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:31,033 INFO [RS:1;asf911:47183] regionserver.HRegionServer(866): ClusterId : 58f2da67-30c0-4081-84ee-cf0400bad60b 2018-01-13 07:26:31,034 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:31,034 INFO [RS:0;asf911:33483] regionserver.HRegionServer(866): ClusterId : 58f2da67-30c0-4081-84ee-cf0400bad60b 2018-01-13 07:26:31,034 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:31,040 DEBUG [RS:1;asf911:47183] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:31,048 DEBUG [RS:0;asf911:33483] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:31,049 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:31,049 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:31,050 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:31,050 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:31,057 DEBUG [RS:1;asf911:47183] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:31,065 DEBUG [RS:0;asf911:33483] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:31,065 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:31,066 DEBUG [RS:1;asf911:47183] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cc09c95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:31,066 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:31,066 DEBUG [RS:1;asf911:47183] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:47183 2018-01-13 07:26:31,066 INFO [RS:1;asf911:47183] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:31,066 INFO [RS:1;asf911:47183] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:31,066 DEBUG [RS:0;asf911:33483] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d00256b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:31,067 DEBUG [RS:0;asf911:33483] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:33483 2018-01-13 07:26:31,067 INFO [RS:0;asf911:33483] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:31,067 INFO [RS:0;asf911:33483] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:31,067 INFO [RS:1;asf911:47183] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,58949,1515828389678 with port=47183, startcode=1515828389778 2018-01-13 07:26:31,068 INFO [RS:0;asf911:33483] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,58949,1515828389678 with port=33483, startcode=1515828389733 2018-01-13 07:26:31,068 DEBUG [RS:1;asf911:47183] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:31,068 DEBUG [RS:0;asf911:33483] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:31,070 INFO [RS-EventLoopGroup-29-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.12 (auth:SIMPLE) 2018-01-13 07:26:31,070 INFO [RS-EventLoopGroup-29-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.13 (auth:SIMPLE) 2018-01-13 07:26:31,071 INFO [RS-EventLoopGroup-29-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 56482 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:31,071 INFO [RS-EventLoopGroup-29-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 44661 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:31,075 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58949] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,076 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=58949] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,076 DEBUG [RS:1;asf911:47183] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:31,076 DEBUG [RS:0;asf911:33483] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:31,077 DEBUG [RS:1;asf911:47183] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:31,077 DEBUG [RS:0;asf911:33483] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:31,077 DEBUG [RS:1;asf911:47183] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:31,077 DEBUG [RS:0;asf911:33483] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:31,082 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:31,082 DEBUG [RS:1;asf911:47183] zookeeper.ZKUtil(355): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,082 DEBUG [RS:0;asf911:33483] zookeeper.ZKUtil(355): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,082 WARN [RS:1;asf911:47183] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:31,082 WARN [RS:0;asf911:33483] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:31,083 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,083 DEBUG [RS:1;asf911:47183] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,083 DEBUG [RS:0;asf911:33483] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,083 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,091 INFO [M:1;asf911:58949] master.ServerManager(891): Finished wait on RegionServer count=2; waited=104ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:31,099 INFO [RS:1;asf911:47183] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:31,099 INFO [RS:0;asf911:33483] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:31,099 INFO [M:1;asf911:58949] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:31,100 DEBUG [RS:1;asf911:47183] zookeeper.ZKUtil(355): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,101 DEBUG [RS:0;asf911:33483] zookeeper.ZKUtil(355): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,101 DEBUG [RS:1;asf911:47183] zookeeper.ZKUtil(355): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,101 DEBUG [RS:0;asf911:33483] zookeeper.ZKUtil(355): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,102 INFO [RS:1;asf911:47183] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x70eda003 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:31,102 INFO [RS:0;asf911:33483] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x75270e21 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:31,122 DEBUG [RS:1;asf911:47183] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79ede626, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:31,122 DEBUG [RS:1;asf911:47183] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:31,122 INFO [RS:1;asf911:47183] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:31,122 DEBUG [RS:0;asf911:33483] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@682d4de0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:31,122 DEBUG [RS:0;asf911:33483] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:31,123 INFO [RS:0;asf911:33483] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:31,126 INFO [RS:1;asf911:47183] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:31,126 INFO [RS:0;asf911:33483] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:31,126 INFO [RS:1;asf911:47183] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:31,126 INFO [RS:0;asf911:33483] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:31,126 INFO [RS:1;asf911:47183] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:31,127 INFO [RS:0;asf911:33483] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:31,134 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:47183, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,134 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:33483, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,135 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:47183, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:31,135 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:33483, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:31,135 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:47183, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,135 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:33483, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,135 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:47183, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,135 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:33483, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,135 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:47183, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:31,135 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:33483, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:31,136 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:47183, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:31,136 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:33483, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:31,136 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:47183, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:31,136 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:33483, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:31,136 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:47183, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,136 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:33483, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:31,136 DEBUG [RS:1;asf911:47183] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:47183, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:31,136 DEBUG [RS:0;asf911:33483] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:33483, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:31,164 INFO [SplitLogWorker-asf911:33483] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,33483,1515828389733 starting 2018-01-13 07:26:31,164 INFO [RS:0;asf911:33483] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:31,169 INFO [RS:1;asf911:47183] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:31,169 INFO [SplitLogWorker-asf911:47183] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,47183,1515828389778 starting 2018-01-13 07:26:31,190 INFO [RS:0;asf911:33483] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,33483,1515828389733, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:33483, sessionid=0x160ee6894500059 2018-01-13 07:26:31,191 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:31,191 DEBUG [RS:0;asf911:33483] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,191 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,33483,1515828389733' 2018-01-13 07:26:31,191 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:31,191 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:31,192 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:31,192 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:31,192 DEBUG [RS:0;asf911:33483] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:31,192 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,33483,1515828389733' 2018-01-13 07:26:31,192 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:31,193 DEBUG [RS:0;asf911:33483] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:31,193 DEBUG [RS:0;asf911:33483] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:31,193 INFO [RS:0;asf911:33483] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:31,193 INFO [RS:0;asf911:33483] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:31,200 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,47183,1515828389778, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:47183, sessionid=0x160ee689450005a 2018-01-13 07:26:31,200 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:31,200 DEBUG [RS:1;asf911:47183] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,202 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,47183,1515828389778' 2018-01-13 07:26:31,202 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:31,203 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:31,204 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:31,204 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:31,204 DEBUG [RS:1;asf911:47183] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,204 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,47183,1515828389778' 2018-01-13 07:26:31,204 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:31,205 DEBUG [RS:1;asf911:47183] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:31,205 DEBUG [RS:1;asf911:47183] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:31,205 INFO [RS:1;asf911:47183] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:31,205 INFO [RS:1;asf911:47183] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:31,279 DEBUG [M:1;asf911:58949] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:31,280 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:31,281 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:31,282 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:31,284 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:26:31,347 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:31,348 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:31,348 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:26:31,349 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:26:31,407 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:31,408 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:26:31,558 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:26:31,558 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:26:31,565 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:31,832 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741888_1066, blk_1073741889_1067, blk_1073741890_1068, blk_1073741891_1069, blk_1073741892_1070, blk_1073741893_1071, blk_1073741894_1072, blk_1073741895_1073, blk_1073741886_1064, blk_1073741887_1065] 2018-01-13 07:26:32,198 INFO [RS:0;asf911:33483] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C33483%2C1515828389733, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:32,208 DEBUG [RS-EventLoopGroup-31-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:32,208 DEBUG [RS-EventLoopGroup-31-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:32,210 INFO [RS:1;asf911:47183] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C47183%2C1515828389778, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:32,211 INFO [RS:0;asf911:33483] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733/asf911.gq1.ygridcore.net%2C33483%2C1515828389733.1515828392199 2018-01-13 07:26:32,212 DEBUG [RS:0;asf911:33483] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:26:32,219 DEBUG [RS-EventLoopGroup-31-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:32,219 DEBUG [RS-EventLoopGroup-31-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:32,223 INFO [RS:1;asf911:47183] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778/asf911.gq1.ygridcore.net%2C47183%2C1515828389778.1515828392211 2018-01-13 07:26:32,223 DEBUG [RS:1;asf911:47183] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:32,226 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,240 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:26:32,270 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,422 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,47183,1515828389778 version=3145728 2018-01-13 07:26:32,423 DEBUG [RSProcedureDispatcher-pool60-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,423 DEBUG [RSProcedureDispatcher-pool60-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:32,425 INFO [RS-EventLoopGroup-31-13] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:32,426 INFO [RS-EventLoopGroup-31-13] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55799 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:32,428 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=47183] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:32,428 INFO [RS_OPEN_META-asf911:47183-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:32,432 INFO [RS_OPEN_META-asf911:47183-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C47183%2C1515828389778.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:32,439 DEBUG [RS-EventLoopGroup-31-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:32,439 DEBUG [RS-EventLoopGroup-31-15] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:32,442 INFO [RS_OPEN_META-asf911:47183-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778/asf911.gq1.ygridcore.net%2C47183%2C1515828389778.meta.1515828392432.meta 2018-01-13 07:26:32,442 DEBUG [RS_OPEN_META-asf911:47183-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:32,443 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:32,443 DEBUG [RS_OPEN_META-asf911:47183-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:32,443 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:32,443 INFO [RS_OPEN_META-asf911:47183-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:32,444 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:32,444 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:32,448 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:32,448 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:32,449 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:32,449 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:32,449 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:32,452 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:32,452 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:32,452 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:32,452 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:32,453 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:32,455 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:32,455 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:32,456 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:32,456 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:32,456 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:32,459 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:32,459 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:32,459 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:32,459 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:32,460 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:32,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:32,462 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:32,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:32,463 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:32,463 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:32,467 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:32,469 DEBUG [RS_OPEN_META-asf911:47183-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:32,474 DEBUG [RS_OPEN_META-asf911:47183-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:32,475 INFO [RS_OPEN_META-asf911:47183-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:26:32,478 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:32,484 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58949] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,485 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:32,485 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828392484}]},"ts":1515828392484} 2018-01-13 07:26:32,487 DEBUG [RS_OPEN_META-asf911:47183-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,492 DEBUG [hconnection-0x4c4224a2-shared-pool63-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:32,494 INFO [RS-EventLoopGroup-31-19] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:32,495 INFO [RS-EventLoopGroup-31-19] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55802 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:32,498 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:32,500 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,515 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:32,530 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:32,572 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:26:32,614 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:32,615 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.1670sec 2018-01-13 07:26:32,656 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.5140sec 2018-01-13 07:26:32,657 INFO [M:1;asf911:58949] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:32,665 INFO [M:1;asf911:58949] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733 belongs to an existing region server 2018-01-13 07:26:32,666 INFO [M:1;asf911:58949] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778 belongs to an existing region server 2018-01-13 07:26:32,666 INFO [M:1;asf911:58949] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:32,668 INFO [M:1;asf911:58949] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:32,668 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828397668 2018-01-13 07:26:32,668 INFO [M:1;asf911:58949] assignment.AssignmentManager(1219): Joined the cluster in 2msec, failover=true 2018-01-13 07:26:32,672 INFO [M:1;asf911:58949] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:26:32,672 INFO [M:1;asf911:58949] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:26:32,814 DEBUG [M:1;asf911:58949] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:32,816 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:32,856 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:32,872 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741904_1082{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:32,872 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741904_1082{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:32,875 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:32,877 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:26:32,890 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741905_1083{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:32,891 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741905_1083{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:32,892 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:32,892 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.: disabling compactions & flushes 2018-01-13 07:26:32,892 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:32,892 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,155 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:33,156 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:33,162 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:33,220 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:33,220 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828393220}]},"ts":1515828393220} 2018-01-13 07:26:33,224 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:26:33,224 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,232 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778}] 2018-01-13 07:26:33,285 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778 hbase:namespace hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,287 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OFFLINE, location=asf911.gq1.ygridcore.net,47183,1515828389778; forceNewPlan=false, retain=false 2018-01-13 07:26:33,440 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:33,443 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,443 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x7129712f to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:33,459 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fee817a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:33,461 DEBUG [MultiHConnection-shared--pool64-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:33,463 INFO [RS-EventLoopGroup-31-21] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:33,464 INFO [RS-EventLoopGroup-31-21] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55815 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:33,467 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,618 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,47183,1515828389778 version=3145728 2018-01-13 07:26:33,621 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=47183] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,627 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] regionserver.HRegion(6949): Opening region: {ENCODED => 4239d8ec9db768700365e5d8b23b3003, NAME => 'hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:33,627 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 4239d8ec9db768700365e5d8b23b3003 2018-01-13 07:26:33,627 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,632 DEBUG [StoreOpener-4239d8ec9db768700365e5d8b23b3003-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/4239d8ec9db768700365e5d8b23b3003/info 2018-01-13 07:26:33,632 DEBUG [StoreOpener-4239d8ec9db768700365e5d8b23b3003-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/4239d8ec9db768700365e5d8b23b3003/info 2018-01-13 07:26:33,633 INFO [StoreOpener-4239d8ec9db768700365e5d8b23b3003-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:33,633 INFO [StoreOpener-4239d8ec9db768700365e5d8b23b3003-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:33,633 INFO [StoreOpener-4239d8ec9db768700365e5d8b23b3003-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:33,635 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/4239d8ec9db768700365e5d8b23b3003 2018-01-13 07:26:33,642 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/4239d8ec9db768700365e5d8b23b3003/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:33,642 INFO [RS_OPEN_PRIORITY_REGION-asf911:47183-0] regionserver.HRegion(960): Onlined 4239d8ec9db768700365e5d8b23b3003; next sequenceid=2 2018-01-13 07:26:33,644 INFO [PostOpenDeployTasks:4239d8ec9db768700365e5d8b23b3003] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,647 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58949] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,647 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,647 DEBUG [PostOpenDeployTasks:4239d8ec9db768700365e5d8b23b3003] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:33,651 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:47183-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. on asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,710 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:33,751 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:26:33,752 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:33,752 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=4239d8ec9db768700365e5d8b23b3003, target=asf911.gq1.ygridcore.net,47183,1515828389778 in 426msec 2018-01-13 07:26:33,752 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828393752}]},"ts":1515828393752} 2018-01-13 07:26:33,756 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:26:33,821 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:33,828 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:26:33,854 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:33,880 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1490sec 2018-01-13 07:26:33,926 DEBUG [M:1;asf911:58949] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:33,928 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:33,976 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:34,034 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:34,088 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:34,104 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:34,163 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:34,226 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 302msec 2018-01-13 07:26:34,393 DEBUG [M:1;asf911:58949] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,395 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,460 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,526 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,594 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,607 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:34,680 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:34,738 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 449msec 2018-01-13 07:26:34,765 DEBUG [M:1;asf911:58949] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:34,773 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:34,782 DEBUG [M:1;asf911:58949] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:34,790 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:34,790 INFO [M:1;asf911:58949] master.HMaster(928): Master has completed initialization 4.943sec 2018-01-13 07:26:34,791 INFO [M:1;asf911:58949] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:34,791 INFO [M:1;asf911:58949] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:34,793 INFO [M:1;asf911:58949] regionserver.HRegionServer(866): ClusterId : 58f2da67-30c0-4081-84ee-cf0400bad60b 2018-01-13 07:26:34,816 DEBUG [hconnection-0x51a18b21-shared-pool12-t7] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:34,818 DEBUG [Default-IPC-NioEventLoopGroup-7-21] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:37733 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:37733 2018-01-13 07:26:34,833 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741888_1066, blk_1073741889_1067, blk_1073741890_1068, blk_1073741891_1069, blk_1073741892_1070, blk_1073741893_1071, blk_1073741894_1072, blk_1073741895_1073, blk_1073741886_1064, blk_1073741887_1065] 2018-01-13 07:26:34,923 DEBUG [hconnection-0x51a18b21-shared-pool12-t7] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:34,926 INFO [RS-EventLoopGroup-31-22] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:34,926 INFO [RS-EventLoopGroup-31-22] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55828 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:34,929 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:34,929 INFO [Time-limited test] hbase.TestZooKeeper(147): Starting testMasterZKSessionRecoveryFailure 2018-01-13 07:26:34,930 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:34,930 ERROR [Time-limited test] helpers.MarkerIgnoringBase(159): Test recovery from zk session expired org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:150) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:34,930 INFO [Time-limited test] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,58949,1515828389678' ***** 2018-01-13 07:26:34,930 INFO [Time-limited test] regionserver.HRegionServer(2172): STOPPED: Stopped by Time-limited test 2018-01-13 07:26:34,932 DEBUG [M:1;asf911:58949] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fd779c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:34,932 INFO [M:1;asf911:58949] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,58949,1515828389678 2018-01-13 07:26:34,932 DEBUG [M:1;asf911:58949] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:34,933 INFO [M:1;asf911:58949] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x23e1792e to localhost:59094 2018-01-13 07:26:34,933 DEBUG [M:1;asf911:58949] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:34,933 INFO [M:1;asf911:58949] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,58949,1515828389678; all regions closed. 2018-01-13 07:26:34,933 DEBUG [M:1;asf911:58949] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:34,934 INFO [M:1;asf911:58949] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,58949,1515828389678 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,58949,1515828389678-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,58949,1515828389678-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:58949 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,58949,1515828389678-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,58949,1515828389678-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,58949,1515828389678-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:34,934 INFO [M:1;asf911:58949] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:34,934 INFO [M:1;asf911:58949] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:34,934 DEBUG [M:1;asf911:58949] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:34,940 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,940 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,940 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,940 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,940 INFO [Time-limited test] hbase.TestZooKeeper(165): Creating table testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:34,941 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:55974 2018-01-13 07:26:34,942 DEBUG [Default-IPC-NioEventLoopGroup-7-23] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:55974 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:55974 2018-01-13 07:26:34,942 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:55974 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:55974 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:55974 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:34,948 DEBUG [M:1;asf911:58949] zookeeper.ZKUtil(614): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:34,950 WARN [M:1;asf911:58949] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:34,950 INFO [M:1;asf911:58949] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:34,950 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:34,950 INFO [M:1;asf911:58949] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x7129712f to localhost:59094 2018-01-13 07:26:34,950 DEBUG [M:1;asf911:58949] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:34,948 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:34,950 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:34,950 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:34,952 INFO [M:1;asf911:58949] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:34,952 INFO [M:1;asf911:58949] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:34,954 INFO [M:1;asf911:58949] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:34,954 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,955 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,957 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:34,958 DEBUG [main-EventThread] master.ActiveMasterManager(132): No master available. Notifying waiting threads 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,956 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:34,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741900_1078{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:34,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741900_1078{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:34,965 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,965 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,965 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,965 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:34,965 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:34,965 INFO [M:1;asf911:58949] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,58949,1515828389678_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:34,966 INFO [M:1;asf911:58949] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:34,966 INFO [M:1;asf911:58949] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:34,966 INFO [M:0;asf911:42935] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,42935,1515828389567 from backup master directory 2018-01-13 07:26:34,966 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:34,966 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:34,973 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:34,973 WARN [M:0;asf911:42935] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:34,973 INFO [M:0;asf911:42935] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:34,982 DEBUG [M:1;asf911:58949] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,58949,1515828389678 already deleted, retry=false 2018-01-13 07:26:34,993 DEBUG [M:0;asf911:42935] util.FSTableDescriptors(787): Current tableInfoPath = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:34,996 INFO [M:1;asf911:58949] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,58949,1515828389678; zookeeper connection closed. 2018-01-13 07:26:34,996 INFO [M:1;asf911:58949] regionserver.HRegionServer(1157): M:1;asf911:58949 exiting 2018-01-13 07:26:35,002 DEBUG [M:0;asf911:42935] util.FSTableDescriptors(791): TableInfo already exists.. Skipping creation 2018-01-13 07:26:35,014 INFO [M:0;asf911:42935] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:35,015 INFO [M:0;asf911:42935] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:35,015 DEBUG [M:0;asf911:42935] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:35,033 INFO [M:0;asf911:42935] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x4a9f2787 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:35,042 DEBUG [M:0;asf911:42935] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b0781c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:35,058 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:55974 this server is in the failed servers list 2018-01-13 07:26:35,059 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:55974 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:55974 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:55974 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:35,060 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42935 2018-01-13 07:26:35,062 INFO [RS-EventLoopGroup-28-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:35,063 INFO [RS-EventLoopGroup-28-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 36495 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:35,066 INFO [M:0;asf911:42935] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:35,066 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:35,067 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:35,068 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:35,069 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:35,069 INFO [M:0;asf911:42935] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:35,070 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:35,071 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:35,073 INFO [M:0;asf911:42935] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,42935,1515828389567, sessionid=0x160ee6894500057, setting cluster-up flag (Was=true) 2018-01-13 07:26:35,082 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:35,083 INFO [M:0;asf911:42935] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:35,084 DEBUG [M:0;asf911:42935] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:35,090 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:35,091 INFO [M:0;asf911:42935] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:35,092 DEBUG [M:0;asf911:42935] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:35,094 INFO [M:0;asf911:42935] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:35,094 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:42935, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:35,094 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:42935, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:35,094 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:42935, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:35,094 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:42935, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:35,095 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:42935, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:35,095 DEBUG [M:0;asf911:42935] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:42935, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:35,097 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:35,097 INFO [M:0;asf911:42935] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:35,098 INFO [M:0;asf911:42935] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:26:35,099 INFO [M:0;asf911:42935] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-01-13 07:26:35,100 DEBUG [M:0;asf911:42935] wal.WALProcedureStore(1286): Opening Pv2 FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=12191; replication=2; blocksize=134217728; modification_time=1515828394965; access_time=1515828390964; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-01-13 07:26:35,111 DEBUG [M:0;asf911:42935] wal.WALProcedureStore(1071): Roll new state log: 2 2018-01-13 07:26:35,113 INFO [M:0;asf911:42935] wal.WALProcedureStore(390): Lease acquired for flushLogId: 2 2018-01-13 07:26:35,113 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 16msec 2018-01-13 07:26:35,118 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor$1(317): Load max pid=6 2018-01-13 07:26:35,119 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor(356): Completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:35,119 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor(356): Completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:35,120 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor(356): Completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:35,120 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor(356): Completed pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:35,121 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 8msec 2018-01-13 07:26:35,123 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828425123 2018-01-13 07:26:35,123 INFO [M:0;asf911:42935] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:35,123 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:35,124 INFO [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:35,124 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:35,124 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:35,125 INFO [M:0;asf911:42935] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:35,127 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:35,127 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:35,128 DEBUG [M:0;asf911:42935] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:35,128 DEBUG [M:0;asf911:42935] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:42935-HFileCleaner.large.0-1515828395128,5,main] 2018-01-13 07:26:35,128 DEBUG [M:0;asf911:42935] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:42935-HFileCleaner.small.0-1515828395128,5,main] 2018-01-13 07:26:35,128 INFO [M:0;asf911:42935] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828389814, completionTime=-1 2018-01-13 07:26:35,128 INFO [M:0;asf911:42935] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:35,132 DEBUG [M:0;asf911:42935-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:35,134 DEBUG [M:0;asf911:42935-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500063 connected 2018-01-13 07:26:35,218 DEBUG [RS:0;asf911:33483] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:35,221 DEBUG [RS-EventLoopGroup-31-23] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:58949 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:35,234 DEBUG [RS:1;asf911:47183] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:35,234 DEBUG [RS-EventLoopGroup-31-24] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:58949 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:58949 2018-01-13 07:26:35,268 DEBUG [Time-limited test] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:55974 this server is in the failed servers list 2018-01-13 07:26:35,268 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:55974 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:55974 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java) at org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1011) at org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:1740) at org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1213) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:105) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3054) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3046) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:645) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:618) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:592) at org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:168) at org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: asf911.gq1.ygridcore.net/67.195.81.155:55974 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:352) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420) ... 28 more 2018-01-13 07:26:35,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] ipc.CallRunner(141): callId: 70 service: MasterService methodName: CreateTable size: 563 connection: 67.195.81.155:36495 deadline: 1515828455272 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2702) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1729) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:592) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:35,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] ipc.CallRunner(141): callId: 72 service: MasterService methodName: CreateTable size: 564 connection: 67.195.81.155:36495 deadline: 1515828455578 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2702) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1729) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:592) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:36,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] ipc.CallRunner(141): callId: 74 service: MasterService methodName: CreateTable size: 564 connection: 67.195.81.155:36495 deadline: 1515828456085 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2702) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1729) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:592) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:36,222 DEBUG [RS:0;asf911:33483] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42935 2018-01-13 07:26:36,225 INFO [RS-EventLoopGroup-28-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.12 (auth:SIMPLE) 2018-01-13 07:26:36,225 INFO [RS-EventLoopGroup-28-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 50927 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:36,225 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:36,231 INFO [M:0;asf911:42935] master.ServerManager(871): Waiting on RegionServer count=1 to settle; waited=1103ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:36,236 DEBUG [RS:1;asf911:47183] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42935 2018-01-13 07:26:36,238 INFO [RS-EventLoopGroup-28-4] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.13 (auth:SIMPLE) 2018-01-13 07:26:36,238 INFO [RS-EventLoopGroup-28-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 54570 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:36,239 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,240 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] assignment.AssignmentManager(996): META REPORTED: rit=OFFLINE, location=null, table=hbase:meta, region=1588230740 2018-01-13 07:26:36,240 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] assignment.AssignmentManager(998): META REPORTED but no procedure found (complete?); set location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,282 INFO [M:0;asf911:42935] master.ServerManager(891): Finished wait on RegionServer count=2; waited=1154ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:36,285 INFO [M:0;asf911:42935] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:36,478 DEBUG [M:0;asf911:42935] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:36,479 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:36,479 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=7, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:36,483 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733 belongs to an existing region server 2018-01-13 07:26:36,484 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778 belongs to an existing region server 2018-01-13 07:26:36,548 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:36,549 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=7, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,555 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778}] 2018-01-13 07:26:36,623 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:36,623 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OFFLINE, location=asf911.gq1.ygridcore.net,47183,1515828389778; forceNewPlan=false, retain=true 2018-01-13 07:26:36,716 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x497029f7 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:36,718 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x5bfa2aba no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:36,779 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:36,784 INFO [ProcExecWrkr-7] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,790 INFO [ProcExecWrkr-7] assignment.RegionTransitionProcedure(208): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,942 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,47183,1515828389778 version=3145728 2018-01-13 07:26:36,943 DEBUG [RSProcedureDispatcher-pool68-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:36,943 DEBUG [RSProcedureDispatcher-pool68-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:36,947 INFO [RS-EventLoopGroup-31-28] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:36,947 INFO [RS-EventLoopGroup-31-28] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55847 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:36,948 WARN [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=47183] regionserver.RSRpcServices(1901): Received OPEN for the region:hbase:meta,,1.1588230740, which is already online 2018-01-13 07:26:37,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] ipc.CallRunner(141): callId: 76 service: MasterService methodName: CreateTable size: 564 connection: 67.195.81.155:36495 deadline: 1515828457097 org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2702) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1729) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:592) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:37,124 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:37,124 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:37,243 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] assignment.AssignmentManager(996): META REPORTED: rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778, table=hbase:meta, region=1588230740 2018-01-13 07:26:37,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=0, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:37,244 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828397243}]},"ts":1515828397243} 2018-01-13 07:26:37,250 DEBUG [hconnection-0x439569a8-shared-pool69-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:37,253 INFO [RS-EventLoopGroup-31-30] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:37,253 INFO [RS-EventLoopGroup-31-30] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55848 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:37,256 INFO [ProcExecWrkr-10] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:37,258 INFO [ProcExecWrkr-10] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:37,331 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:37,389 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:37,390 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,47183,1515828389778 in 724msec 2018-01-13 07:26:37,439 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.1040sec 2018-01-13 07:26:37,440 INFO [M:0;asf911:42935] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:37,456 INFO [M:0;asf911:42935] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733 belongs to an existing region server 2018-01-13 07:26:37,457 INFO [M:0;asf911:42935] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778 belongs to an existing region server 2018-01-13 07:26:37,457 INFO [M:0;asf911:42935] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:37,460 INFO [M:0;asf911:42935] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 4239d8ec9db768700365e5d8b23b3003, NAME => 'hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.', STARTKEY => '', ENDKEY => ''} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,47183,1515828389778 regionLocation=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:37,462 INFO [M:0;asf911:42935] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:37,462 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828402462 2018-01-13 07:26:37,463 INFO [M:0;asf911:42935] assignment.AssignmentManager(1219): Joined the cluster in 6msec, failover=true 2018-01-13 07:26:37,467 DEBUG [M:0;asf911:42935] zookeeper.ZKUtil(355): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:37,490 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:37,498 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:37,513 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:37,521 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:37,521 INFO [M:0;asf911:42935] master.HMaster(928): Master has completed initialization 2.547sec 2018-01-13 07:26:37,522 INFO [M:0;asf911:42935] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:37,522 INFO [M:0;asf911:42935] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:37,523 INFO [M:0;asf911:42935] regionserver.HRegionServer(866): ClusterId : 58f2da67-30c0-4081-84ee-cf0400bad60b 2018-01-13 07:26:37,952 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:39,108 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.HMaster$3(1743): Client=jenkins//67.195.81.155 create 'testMasterZKSessionRecoveryFailure_1515828394940', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-13 07:26:39,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] procedure2.ProcedureExecutor(869): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,275 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:39,328 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,341 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741907_1085{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:39,342 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741907_1085 size 517 2018-01-13 07:26:39,344 DEBUG [ProcExecWrkr-8] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/default/testMasterZKSessionRecoveryFailure_1515828394940/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:39,346 INFO [RegionOpenAndInitThread-testMasterZKSessionRecoveryFailure_1515828394940-1] regionserver.HRegion(6772): creating HRegion testMasterZKSessionRecoveryFailure_1515828394940 HTD == 'testMasterZKSessionRecoveryFailure_1515828394940', {NAME => 'fam', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,359 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741908_1086{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:39,360 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741908_1086{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:39,361 DEBUG [RegionOpenAndInitThread-testMasterZKSessionRecoveryFailure_1515828394940-1] regionserver.HRegion(807): Instantiated testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:39,361 DEBUG [RegionOpenAndInitThread-testMasterZKSessionRecoveryFailure_1515828394940-1] regionserver.HRegion(1481): Closing testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.: disabling compactions & flushes 2018-01-13 07:26:39,361 DEBUG [RegionOpenAndInitThread-testMasterZKSessionRecoveryFailure_1515828394940-1] regionserver.HRegion(1520): Updates disabled for region testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:39,361 INFO [RegionOpenAndInitThread-testMasterZKSessionRecoveryFailure_1515828394940-1] regionserver.HRegion(1633): Closed testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:39,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:39,474 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,476 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.","families":{"info":[{"qualifier":"regioninfo","vlen":82,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:39,480 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:39,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:39,609 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:39,609 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMasterZKSessionRecoveryFailure_1515828394940","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828399609}]},"ts":1515828399609} 2018-01-13 07:26:39,614 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1786): Updated table testMasterZKSessionRecoveryFailure_1515828394940 state to ENABLING in META 2018-01-13 07:26:39,614 DEBUG [ProcExecWrkr-8] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:39,621 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778}] 2018-01-13 07:26:39,753 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(883): pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778 testMasterZKSessionRecoveryFailure_1515828394940 testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:39,755 INFO [ProcExecWrkr-8] assignment.AssignProcedure(211): Start pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OFFLINE, location=asf911.gq1.ygridcore.net,47183,1515828389778; forceNewPlan=false, retain=false 2018-01-13 07:26:39,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:39,907 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:39,916 INFO [ProcExecWrkr-4] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:39,917 INFO [ProcExecWrkr-4] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x021096e1 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:39,934 DEBUG [ProcExecWrkr-4] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cd768d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:39,935 DEBUG [MultiHConnection-shared--pool73-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:39,937 INFO [RS-EventLoopGroup-31-32] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:39,938 INFO [RS-EventLoopGroup-31-32] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 55882 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:39,950 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:40,101 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,47183,1515828389778 version=3145728 2018-01-13 07:26:40,104 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=47183] regionserver.RSRpcServices(1907): Open testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:40,110 DEBUG [RS_OPEN_REGION-asf911:47183-0] regionserver.HRegion(6949): Opening region: {ENCODED => 9c99aa777bba008abbae573de490d240, NAME => 'testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:40,110 DEBUG [RS_OPEN_REGION-asf911:47183-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMasterZKSessionRecoveryFailure_1515828394940 9c99aa777bba008abbae573de490d240 2018-01-13 07:26:40,110 DEBUG [RS_OPEN_REGION-asf911:47183-0] regionserver.HRegion(807): Instantiated testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:40,116 DEBUG [StoreOpener-9c99aa777bba008abbae573de490d240-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterZKSessionRecoveryFailure_1515828394940/9c99aa777bba008abbae573de490d240/fam 2018-01-13 07:26:40,116 DEBUG [StoreOpener-9c99aa777bba008abbae573de490d240-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterZKSessionRecoveryFailure_1515828394940/9c99aa777bba008abbae573de490d240/fam 2018-01-13 07:26:40,116 INFO [StoreOpener-9c99aa777bba008abbae573de490d240-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:40,117 INFO [StoreOpener-9c99aa777bba008abbae573de490d240-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:40,117 INFO [StoreOpener-9c99aa777bba008abbae573de490d240-1] hfile.CacheConfig(239): Created cacheConfig for fam: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:40,117 INFO [StoreOpener-9c99aa777bba008abbae573de490d240-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:40,118 INFO [RS_OPEN_REGION-asf911:47183-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:40,119 DEBUG [RS_OPEN_REGION-asf911:47183-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterZKSessionRecoveryFailure_1515828394940/9c99aa777bba008abbae573de490d240 2018-01-13 07:26:40,122 DEBUG [RS_OPEN_REGION-asf911:47183-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMasterZKSessionRecoveryFailure_1515828394940 descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:40,126 DEBUG [RS_OPEN_REGION-asf911:47183-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testMasterZKSessionRecoveryFailure_1515828394940/9c99aa777bba008abbae573de490d240/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:40,126 INFO [RS_OPEN_REGION-asf911:47183-0] regionserver.HRegion(960): Onlined 9c99aa777bba008abbae573de490d240; next sequenceid=2 2018-01-13 07:26:40,128 INFO [PostOpenDeployTasks:9c99aa777bba008abbae573de490d240] regionserver.HRegionServer(2197): Post open deploy tasks for testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:40,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42935] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778; rit=OPENING, location=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:40,135 INFO [ProcExecWrkr-3] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:40,135 DEBUG [PostOpenDeployTasks:9c99aa777bba008abbae573de490d240] regionserver.HRegionServer(2221): Finished post open deploy task for testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:40,138 DEBUG [RS_OPEN_REGION-asf911:47183-0] handler.OpenRegionHandler(128): Opened testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. on asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:40,222 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:40,300 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940; resume parent processing. 2018-01-13 07:26:40,300 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:40,300 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMasterZKSessionRecoveryFailure_1515828394940, region=9c99aa777bba008abbae573de490d240, target=asf911.gq1.ygridcore.net,47183,1515828389778 in 525msec 2018-01-13 07:26:40,301 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMasterZKSessionRecoveryFailure_1515828394940","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828400300}]},"ts":1515828400300} 2018-01-13 07:26:40,305 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table testMasterZKSessionRecoveryFailure_1515828394940 state to ENABLED in META 2018-01-13 07:26:40,369 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:40,424 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMasterZKSessionRecoveryFailure_1515828394940 in 1.2600sec 2018-01-13 07:26:40,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:40,434 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-13 07:26:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.MasterRpcServices(1157): Checking to see if procedure is done pid=9 2018-01-13 07:26:41,391 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3638): Operation: CREATE, Table Name: default:testMasterZKSessionRecoveryFailure_1515828394940 completed 2018-01-13 07:26:41,391 INFO [Time-limited test] hbase.TestZooKeeper(176): Putting table testMasterZKSessionRecoveryFailure_1515828394940 2018-01-13 07:26:41,403 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,33483,1515828389733: killAll ***** 2018-01-13 07:26:41,403 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:41,408 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 620765288 }, "NonHeapMemoryUsage" : { "committed" : 134152192, "init" : 2555904, "max" : -1, "used" : 132020704 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 52847, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 251, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 2, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 7, "TotalCallTime_99th_percentile" : 7, "TotalCallTime_99.9th_percentile" : 7, "TotalCallTime_TimeRangeCount_0-1" : 26, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_3-10" : 2, "TotalCallTime_TimeRangeCount_600000-inf" : 213, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 251, "ResponseSize_min" : 0, "ResponseSize_max" : 872, "ResponseSize_mean" : 134, "ResponseSize_25th_percentile" : 5, "ResponseSize_median" : 47, "ResponseSize_75th_percentile" : 57, "ResponseSize_90th_percentile" : 765, "ResponseSize_95th_percentile" : 767, "ResponseSize_98th_percentile" : 871, "ResponseSize_99th_percentile" : 871, "ResponseSize_99.9th_percentile" : 871, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 14, "ResponseSize_SizeRangeCount_100-1000" : 8, "ResponseSize_SizeRangeCount_100000000-inf" : 213, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 251, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 7, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 2, "ProcessCallTime_95th_percentile" : 5, "ProcessCallTime_98th_percentile" : 7, "ProcessCallTime_99th_percentile" : 7, "ProcessCallTime_99.9th_percentile" : 7, "ProcessCallTime_TimeRangeCount_0-1" : 27, "ProcessCallTime_TimeRangeCount_1-3" : 9, "ProcessCallTime_TimeRangeCount_3-10" : 2, "ProcessCallTime_TimeRangeCount_600000-inf" : 213, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 41, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 251, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 133, "RequestSize_25th_percentile" : 87, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 122, "RequestSize_90th_percentile" : 151, "RequestSize_95th_percentile" : 475, "RequestSize_98th_percentile" : 632, "RequestSize_99th_percentile" : 684, "RequestSize_99.9th_percentile" : 731, "RequestSize_SizeRangeCount_10-100" : 20, "RequestSize_SizeRangeCount_100-1000" : 18, "RequestSize_SizeRangeCount_100000000-inf" : 213, "sentBytes" : 26783, "QueueCallTime_num_ops" : 251, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 38, "QueueCallTime_TimeRangeCount_600000-inf" : 213, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:41,413 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,33483,1515828389733 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,33483,1515828389733: killAll ***** 2018-01-13 07:26:41,413 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,33483,1515828389733' ***** 2018-01-13 07:26:41,414 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:41,414 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,47183,1515828389778: killAll ***** 2018-01-13 07:26:41,414 INFO [RS:0;asf911:33483] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:41,414 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:41,414 INFO [SplitLogWorker-asf911:33483] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:41,414 INFO [RS:0;asf911:33483] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:41,415 INFO [SplitLogWorker-asf911:33483] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,33483,1515828389733 exiting 2018-01-13 07:26:41,415 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:41,416 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:41,416 INFO [RS:0;asf911:33483] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:41,417 INFO [RS:0;asf911:33483] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:41,417 INFO [RS:0;asf911:33483] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:41,417 DEBUG [RS:0;asf911:33483] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:41,417 INFO [RS:0;asf911:33483] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x47f84e1b to localhost:59094 2018-01-13 07:26:41,417 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 620789888 }, "NonHeapMemoryUsage" : { "committed" : 134152192, "init" : 2555904, "max" : -1, "used" : 132020736 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 52847, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 251, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7, "TotalCallTime_mean" : 1, "TotalCallTime_25th_percentile" : 0, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 2, "TotalCallTime_95th_percentile" : 5, "TotalCallTime_98th_percentile" : 7, "TotalCallTime_99th_percentile" : 7, "TotalCallTime_99.9th_percentile" : 7, "TotalCallTime_TimeRangeCount_0-1" : 26, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_3-10" : 2, "TotalCallTime_TimeRangeCount_600000-inf" : 213, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 251, "ResponseSize_min" : 0, "ResponseSize_max" : 872, "ResponseSize_mean" : 134, "ResponseSize_25th_percentile" : 5, "ResponseSize_median" : 47, "ResponseSize_75th_percentile" : 57, "ResponseSize_90th_percentile" : 765, "ResponseSize_95th_percentile" : 767, "ResponseSize_98th_percentile" : 871, "ResponseSize_99th_percentile" : 871, "ResponseSize_99.9th_percentile" : 871, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 14, "ResponseSize_SizeRangeCount_100-1000" : 8, "ResponseSize_SizeRangeCount_100000000-inf" : 213, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 251, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 7, "ProcessCallTime_mean" : 1, "ProcessCallTime_25th_percentile" : 0, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 2, "ProcessCallTime_95th_percentile" : 5, "ProcessCallTime_98th_percentile" : 7, "ProcessCallTime_99th_percentile" : 7, "ProcessCallTime_99.9th_percentile" : 7, "ProcessCallTime_TimeRangeCount_0-1" : 27, "ProcessCallTime_TimeRangeCount_1-3" : 9, "ProcessCallTime_TimeRangeCount_3-10" : 2, "ProcessCallTime_TimeRangeCount_600000-inf" : 213, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 41, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 251, "RequestSize_min" : 70, "RequestSize_max" : 737, "RequestSize_mean" : 133, "RequestSize_25th_percentile" : 87, "RequestSize_median" : 88, "RequestSize_75th_percentile" : 122, "RequestSize_90th_percentile" : 151, "RequestSize_95th_percentile" : 475, "RequestSize_98th_percentile" : 632, "RequestSize_99th_percentile" : 684, "RequestSize_99.9th_percentile" : 731, "RequestSize_SizeRangeCount_10-100" : 20, "RequestSize_SizeRangeCount_100-1000" : 18, "RequestSize_SizeRangeCount_100000000-inf" : 213, "sentBytes" : 26783, "QueueCallTime_num_ops" : 251, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 38, "QueueCallTime_TimeRangeCount_600000-inf" : 213, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:41,417 DEBUG [RS:0;asf911:33483] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,419 INFO [RS:0;asf911:33483] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,33483,1515828389733; all regions closed. 2018-01-13 07:26:41,420 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=42935] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,47183,1515828389778 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,47183,1515828389778: killAll ***** 2018-01-13 07:26:41,420 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,47183,1515828389778' ***** 2018-01-13 07:26:41,420 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:41,420 INFO [RS:1;asf911:47183] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:41,420 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:41,421 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:41,421 INFO [RS:1;asf911:47183] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:41,421 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,42935,1515828389567' ***** 2018-01-13 07:26:41,421 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:41,421 INFO [SplitLogWorker-asf911:47183] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:41,423 INFO [SplitLogWorker-asf911:47183] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,47183,1515828389778 exiting 2018-01-13 07:26:41,423 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:41,421 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:41,421 INFO [RS:1;asf911:47183] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:41,423 INFO [RS:1;asf911:47183] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:41,423 DEBUG [M:0;asf911:42935] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@741752dd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:41,424 INFO [M:0;asf911:42935] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:41,424 DEBUG [RS_CLOSE_REGION-asf911:47183-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:41,424 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:41,424 DEBUG [RS:1;asf911:47183] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:41,424 DEBUG [M:0;asf911:42935] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:41,426 INFO [RS:1;asf911:47183] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5638f7e5 to localhost:59094 2018-01-13 07:26:41,426 DEBUG [RS_CLOSE_REGION-asf911:47183-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.: disabling compactions & flushes 2018-01-13 07:26:41,426 INFO [M:0;asf911:42935] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x4a9f2787 to localhost:59094 2018-01-13 07:26:41,424 DEBUG [RS_CLOSE_REGION-asf911:47183-1] handler.CloseRegionHandler(92): Processing close of testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:41,426 DEBUG [M:0;asf911:42935] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,426 DEBUG [RS_CLOSE_REGION-asf911:47183-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:41,426 DEBUG [RS:1;asf911:47183] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,426 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,33483,1515828389733/asf911.gq1.ygridcore.net%2C33483%2C1515828389733.1515828392199 not finished, retry = 0 2018-01-13 07:26:41,426 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741901_1079{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 96 2018-01-13 07:26:41,428 INFO [RS:1;asf911:47183] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:41,428 INFO [RS:1;asf911:47183] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:41,429 INFO [RS:1;asf911:47183] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:41,428 INFO [M:0;asf911:42935] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,42935,1515828389567; all regions closed. 2018-01-13 07:26:41,429 DEBUG [M:0;asf911:42935] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,428 DEBUG [RS_CLOSE_REGION-asf911:47183-1] regionserver.HRegion(1481): Closing testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.: disabling compactions & flushes 2018-01-13 07:26:41,429 INFO [M:0;asf911:42935] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,42935,1515828389567 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42935,1515828389567-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42935,1515828389567-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42935,1515828389567-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:42935 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42935,1515828389567-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,42935,1515828389567-MobCompactionChore Period: 604800 Unit: SECONDS]] on shutdown 2018-01-13 07:26:41,429 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1388): Waiting on 3 regions to close 2018-01-13 07:26:41,429 DEBUG [RS_CLOSE_META-asf911:47183-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:41,428 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741901_1079 size 96 2018-01-13 07:26:41,428 INFO [StoreCloserThread-hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003.-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:41,430 DEBUG [RS_CLOSE_META-asf911:47183-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:41,429 DEBUG [RS:1;asf911:47183] regionserver.HRegionServer(1392): Online Regions={4239d8ec9db768700365e5d8b23b3003=hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003., 1588230740=hbase:meta,,1.1588230740, 9c99aa777bba008abbae573de490d240=testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.} 2018-01-13 07:26:41,429 INFO [M:0;asf911:42935] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:41,431 INFO [M:0;asf911:42935] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:41,431 DEBUG [M:0;asf911:42935] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:41,429 DEBUG [RS_CLOSE_REGION-asf911:47183-1] regionserver.HRegion(1520): Updates disabled for region testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:41,431 ERROR [RS_CLOSE_REGION-asf911:47183-0] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:26:41,431 DEBUG [RS_CLOSE_META-asf911:47183-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:41,433 INFO [RS_CLOSE_REGION-asf911:47183-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:41,433 INFO [StoreCloserThread-testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240.-1] regionserver.HStore(930): Closed fam 2018-01-13 07:26:41,434 DEBUG [RS_CLOSE_REGION-asf911:47183-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828392672.4239d8ec9db768700365e5d8b23b3003. 2018-01-13 07:26:41,434 ERROR [RS_CLOSE_REGION-asf911:47183-1] regionserver.HRegion(1620): Memstore size is 41 2018-01-13 07:26:41,434 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:41,435 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:41,435 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:41,435 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:41,435 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:41,435 INFO [RS_CLOSE_REGION-asf911:47183-1] regionserver.HRegion(1633): Closed testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:41,436 ERROR [RS_CLOSE_META-asf911:47183-0] regionserver.HRegion(1620): Memstore size is 3179 2018-01-13 07:26:41,436 DEBUG [RS_CLOSE_REGION-asf911:47183-1] handler.CloseRegionHandler(125): Closed testMasterZKSessionRecoveryFailure_1515828394940,,1515828399107.9c99aa777bba008abbae573de490d240. 2018-01-13 07:26:41,436 DEBUG [RS_CLOSE_META-asf911:47183-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:41,437 INFO [RS_CLOSE_META-asf911:47183-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:41,438 DEBUG [RS_CLOSE_META-asf911:47183-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:41,440 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:41,440 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:41,448 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:26:41,446 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:41,449 WARN [main] zookeeper.ZKUtil(618): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:41,449 ERROR [main] zookeeper.ZKWatcher(578): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:41,450 ERROR [main] master.ActiveMasterManager(280): master:58949-0x160ee6894500058, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:41,450 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,42935,1515828389567 2018-01-13 07:26:41,457 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:41,457 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:41,457 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(176): Node /hbase/master already deleted, retry=false 2018-01-13 07:26:41,457 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:41,457 ERROR [M:0;asf911:42935] master.ActiveMasterManager(280): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:274) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:41,458 INFO [M:0;asf911:42935] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:41,458 INFO [M:0;asf911:42935] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x021096e1 to localhost:59094 2018-01-13 07:26:41,458 DEBUG [M:0;asf911:42935] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,458 INFO [M:0;asf911:42935] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:41,459 INFO [M:0;asf911:42935] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:41,460 INFO [M:0;asf911:42935] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,462 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,463 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:41,471 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741906_1084{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:41,472 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741906_1084 size 10047 2018-01-13 07:26:41,472 INFO [M:0;asf911:42935] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,42935,1515828389567_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:41,473 INFO [M:0;asf911:42935] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:41,473 INFO [M:0;asf911:42935] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:41,490 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:41,490 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:41,490 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:41,498 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:41,499 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:41,499 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:42935-0x160ee6894500057, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:41,499 DEBUG [M:0;asf911:42935] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,42935,1515828389567 already deleted, retry=false 2018-01-13 07:26:41,529 INFO [M:0;asf911:42935] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,42935,1515828389567; zookeeper connection closed. 2018-01-13 07:26:41,529 INFO [M:0;asf911:42935] regionserver.HRegionServer(1157): M:0;asf911:42935 exiting 2018-01-13 07:26:41,531 DEBUG [RS:0;asf911:33483] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,531 INFO [RS:0;asf911:33483] regionserver.Leases(147): RS:0;asf911:33483 closing leases 2018-01-13 07:26:41,531 INFO [RS:0;asf911:33483] regionserver.Leases(150): RS:0;asf911:33483 closed leases 2018-01-13 07:26:41,531 INFO [RS:0;asf911:33483] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,33483,1515828389733 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,33483,1515828389733 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:41,532 INFO [RS:0;asf911:33483] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:41,532 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:41,532 INFO [RS:0;asf911:33483] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:41,532 INFO [RS:0;asf911:33483] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:41,533 INFO [RS:0;asf911:33483] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:41,548 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:41,548 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,33483,1515828389733 2018-01-13 07:26:41,548 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:41,549 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:33483-0x160ee6894500059, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:41,557 INFO [RS:0;asf911:33483] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,33483,1515828389733; zookeeper connection closed. 2018-01-13 07:26:41,557 INFO [RS:0;asf911:33483] regionserver.HRegionServer(1157): RS:0;asf911:33483 exiting 2018-01-13 07:26:41,557 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@72b4971c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@72b4971c 2018-01-13 07:26:41,631 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,47183,1515828389778; all regions closed. 2018-01-13 07:26:41,635 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741903_1081{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 4093 2018-01-13 07:26:41,635 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,47183,1515828389778/asf911.gq1.ygridcore.net%2C47183%2C1515828389778.meta.1515828392432.meta not finished, retry = 0 2018-01-13 07:26:41,635 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741903_1081 size 4093 2018-01-13 07:26:41,743 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741902_1080{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:41,743 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741902_1080{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:41,744 DEBUG [RS:1;asf911:47183] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:41,744 INFO [RS:1;asf911:47183] regionserver.Leases(147): RS:1;asf911:47183 closing leases 2018-01-13 07:26:41,744 INFO [RS:1;asf911:47183] regionserver.Leases(150): RS:1;asf911:47183 closed leases 2018-01-13 07:26:41,745 INFO [RS:1;asf911:47183] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,47183,1515828389778 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,47183,1515828389778 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:41,745 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:41,745 INFO [RS:1;asf911:47183] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:41,757 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47183-0x160ee689450005a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,47183,1515828389778 2018-01-13 07:26:41,765 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,47183,1515828389778; zookeeper connection closed. 2018-01-13 07:26:41,765 INFO [RS:1;asf911:47183] regionserver.HRegionServer(1157): RS:1;asf911:47183 exiting 2018-01-13 07:26:41,766 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@548fe911] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@548fe911 2018-01-13 07:26:41,766 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 2 regionserver(s) complete 2018-01-13 07:26:41,791 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:41,793 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741900_1078 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,793 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741906_1084 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741901_1079 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741902_1080 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741903_1081 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741907_1085 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741908_1086 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,794 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741899_1077 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,795 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741898_1076 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,795 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741904_1082 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,795 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741905_1083 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:41,795 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741897_1075 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,795 INFO [IPC Server handler 5 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741896_1074 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:41,796 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:41,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:41,808 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500065 connected 2018-01-13 07:26:41,809 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:26:41,810 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:26:41,810 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:26:41,811 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:26:41,812 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:26:41,812 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:26:41,813 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:26:41,813 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:26:41,814 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:26:41,815 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:26:41,815 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:41,816 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:26:41,816 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:26:41,819 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:26:41,819 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:26:41,820 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:26:41,821 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:26:41,822 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:26:41,822 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:26:41,824 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:26:41,824 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:26:41,885 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:41,885 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:41,885 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:41,885 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:26:41,885 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:26:41,886 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:26:41,887 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:41,894 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:41,895 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:41,895 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500065, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:26:41,929 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=522 (was 463) Potentially hanging thread: RS-EventLoopGroup-31-32 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-23 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.13 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-31-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42935-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-31-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.12@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42935-HFileCleaner.small.0-1515828395128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-31-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42935-HFileCleaner.large.0-1515828395128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:95) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-28-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-29 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-28-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-24 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:95) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.12 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-26 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-27 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-28-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:58949-HFileCleaner.small.0-1515828390982 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-31-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:58949-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-31-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:58949-HFileCleaner.large.0-1515828390981 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-31-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:42935-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-31-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-29-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:58949-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1906657932_1 at /127.0.0.1:55100 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-31 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-23 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.13@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-29-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-29-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-21 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-30 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-22 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-28 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-28-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-24 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-31-25 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-30-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=3629 (was 3252) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=338 (was 328) - SystemLoadAverage LEAK? -, ProcessCount=270 (was 270), AvailableMemoryMB=10228 (was 10348) 2018-01-13 07:26:41,929 WARN [main] hbase.ResourceChecker(135): Thread=522 is superior to 500 2018-01-13 07:26:41,929 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=3629 is superior to 1024 2018-01-13 07:26:41,965 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=521, OpenFileDescriptor=3629, MaxFileDescriptor=60000, SystemLoadAverage=338, ProcessCount=270, AvailableMemoryMB=10227 2018-01-13 07:26:41,966 WARN [main] hbase.ResourceChecker(135): Thread=521 is superior to 500 2018-01-13 07:26:41,966 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=3629 is superior to 1024 2018-01-13 07:26:41,981 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741909_1087{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:41,981 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741909_1087 size 7 2018-01-13 07:26:41,983 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:26:41,983 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:26:41,986 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:41,986 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:41,986 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:41,987 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:41,987 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:41,988 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:44597 2018-01-13 07:26:41,988 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:41,988 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:41,989 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:41,990 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:41,991 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:44597 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:42,015 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:445970x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:42,017 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:44597-0x160ee6894500066 connected 2018-01-13 07:26:42,107 DEBUG [main] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:42,108 DEBUG [main] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:42,108 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=44597 2018-01-13 07:26:42,109 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=44597 2018-01-13 07:26:42,110 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=44597 2018-01-13 07:26:42,110 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:42,112 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:42,113 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:42,113 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:42,113 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:42,113 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:42,114 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:40421 2018-01-13 07:26:42,114 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,115 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,115 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,117 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,118 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:40421 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:42,124 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:404210x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:42,125 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:40421-0x160ee6894500067 connected 2018-01-13 07:26:42,132 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:26:42,136 DEBUG [main] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:42,136 DEBUG [main] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:42,137 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=40421 2018-01-13 07:26:42,137 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=40421 2018-01-13 07:26:42,138 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=40421 2018-01-13 07:26:42,139 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:42,139 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:42,139 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:42,141 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:42,141 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:42,165 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:42,165 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:42,166 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:42,166 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:42,166 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:42,166 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:42,167 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:36099 2018-01-13 07:26:42,168 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,168 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,169 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,170 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,171 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:36099 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:42,188 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:360990x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:42,190 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:36099-0x160ee6894500068 connected 2018-01-13 07:26:42,190 DEBUG [main] zookeeper.ZKUtil(357): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:42,191 DEBUG [main] zookeeper.ZKUtil(357): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:42,191 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=36099 2018-01-13 07:26:42,192 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=36099 2018-01-13 07:26:42,194 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=36099 2018-01-13 07:26:42,218 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:42,219 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:42,219 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:42,219 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:42,219 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:42,219 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:42,220 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:49237 2018-01-13 07:26:42,221 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,221 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,222 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,223 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,224 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:49237 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:42,240 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:492370x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:42,242 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:49237-0x160ee6894500069 connected 2018-01-13 07:26:42,243 DEBUG [main] zookeeper.ZKUtil(357): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:42,244 DEBUG [main] zookeeper.ZKUtil(357): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:42,244 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=49237 2018-01-13 07:26:42,245 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=49237 2018-01-13 07:26:42,246 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=49237 2018-01-13 07:26:42,247 INFO [M:0;asf911:44597] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:42,247 INFO [M:1;asf911:40421] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:42,282 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:42,282 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:42,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:42,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:42,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:42,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:42,298 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:26:42,298 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:42,304 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:42,304 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:42,304 INFO [M:0;asf911:44597] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,44597,1515828401985 from backup master directory 2018-01-13 07:26:42,304 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:42,304 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:42,304 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:42,304 INFO [M:1;asf911:40421] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,44597,1515828401985; waiting to become the next active master 2018-01-13 07:26:42,313 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:42,313 WARN [M:0;asf911:44597] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:42,313 INFO [M:0;asf911:44597] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:42,335 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741910_1088{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:42,335 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741910_1088{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:42,337 DEBUG [M:0;asf911:44597] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:42,342 INFO [M:0;asf911:44597] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:26:42,343 INFO [M:0;asf911:44597] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:26:42,366 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741911_1089{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:42,367 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741911_1089 size 32 2018-01-13 07:26:42,367 DEBUG [M:0;asf911:44597] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:42,372 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:42,372 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:42,372 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,372 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:42,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:42,376 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:42,376 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:42,380 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:42,380 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:42,380 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,380 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:42,383 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:42,383 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:42,383 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,383 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:42,386 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:42,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:42,386 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:42,387 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:42,388 DEBUG [M:0;asf911:44597] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:42,391 DEBUG [M:0;asf911:44597] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:42,404 DEBUG [M:0;asf911:44597] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:42,404 INFO [M:0;asf911:44597] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:26:42,404 DEBUG [M:0;asf911:44597] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:42,404 DEBUG [M:0;asf911:44597] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:42,405 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:42,405 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:42,405 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:42,405 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:42,405 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:42,405 INFO [M:0;asf911:44597] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:42,417 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741912_1090{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 2318 2018-01-13 07:26:42,418 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741912_1090 size 2318 2018-01-13 07:26:42,821 DEBUG [M:0;asf911:44597] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:42,839 INFO [M:0;asf911:44597] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:42,840 INFO [M:0;asf911:44597] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:42,840 DEBUG [M:0;asf911:44597] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:42,858 INFO [M:0;asf911:44597] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x08855668 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:42,875 DEBUG [M:0;asf911:44597] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12a888ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:42,896 INFO [M:0;asf911:44597] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:42,896 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:42,897 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:42,924 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:42,925 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:42,925 INFO [M:0;asf911:44597] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:42,948 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:42,948 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:42,948 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:42,948 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:42,949 INFO [M:0;asf911:44597] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,44597,1515828401985, sessionid=0x160ee6894500066, setting cluster-up flag (Was=false) 2018-01-13 07:26:43,015 INFO [M:0;asf911:44597] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:43,016 DEBUG [M:0;asf911:44597] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:43,057 INFO [M:0;asf911:44597] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:43,058 DEBUG [M:0;asf911:44597] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:43,060 INFO [M:0;asf911:44597] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:43,060 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:44597, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:43,060 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:44597, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:43,060 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:44597, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:43,061 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:44597, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:43,061 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:44597, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:43,061 DEBUG [M:0;asf911:44597] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:44597, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:43,063 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:43,063 INFO [M:0;asf911:44597] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:43,065 DEBUG [M:0;asf911:44597] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:26:43,066 INFO [M:0;asf911:44597] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:26:43,066 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 3msec 2018-01-13 07:26:43,067 DEBUG [M:0;asf911:44597] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:26:43,067 DEBUG [M:0;asf911:44597] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:26:43,067 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:26:43,069 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828433069 2018-01-13 07:26:43,069 INFO [M:0;asf911:44597] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:43,069 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:43,070 INFO [M:0;asf911:44597] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:43,070 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:43,071 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:43,071 INFO [M:0;asf911:44597] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:43,073 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:43,073 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:43,074 DEBUG [M:0;asf911:44597] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:43,074 DEBUG [M:0;asf911:44597] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:44597-HFileCleaner.large.0-1515828403074,5,main] 2018-01-13 07:26:43,074 DEBUG [M:0;asf911:44597] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:44597-HFileCleaner.small.0-1515828403074,5,main] 2018-01-13 07:26:43,074 INFO [M:0;asf911:44597] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828402282, completionTime=-1 2018-01-13 07:26:43,075 INFO [M:0;asf911:44597] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:43,080 DEBUG [M:0;asf911:44597-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:43,081 DEBUG [M:0;asf911:44597-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee689450006b connected 2018-01-13 07:26:43,155 INFO [RS:0;asf911:36099] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x0db9783d to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:43,155 INFO [RS:1;asf911:49237] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x294963b3 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:43,183 DEBUG [RS:0;asf911:36099] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37e87fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:43,183 DEBUG [RS:1;asf911:49237] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@278e98ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:43,184 INFO [RS:0;asf911:36099] regionserver.HRegionServer(866): ClusterId : e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:43,184 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:43,184 INFO [RS:1;asf911:49237] regionserver.HRegionServer(866): ClusterId : e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:43,185 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:43,207 DEBUG [RS:0;asf911:36099] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:43,215 DEBUG [RS:1;asf911:49237] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:43,216 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:43,216 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:43,216 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:43,216 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:43,223 DEBUG [RS:0;asf911:36099] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:43,232 DEBUG [RS:1;asf911:49237] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:43,232 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:43,233 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:43,233 DEBUG [RS:0;asf911:36099] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@741134d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:43,233 DEBUG [RS:1;asf911:49237] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23b108c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:43,233 DEBUG [RS:0;asf911:36099] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:36099 2018-01-13 07:26:43,234 INFO [RS:0;asf911:36099] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:43,234 INFO [RS:0;asf911:36099] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:43,234 DEBUG [RS:1;asf911:49237] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:49237 2018-01-13 07:26:43,234 INFO [RS:1;asf911:49237] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:43,234 INFO [RS:1;asf911:49237] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:43,234 INFO [RS:0;asf911:36099] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,44597,1515828401985 with port=36099, startcode=1515828402164 2018-01-13 07:26:43,235 DEBUG [RS:0;asf911:36099] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:43,236 INFO [RS:1;asf911:49237] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,44597,1515828401985 with port=49237, startcode=1515828402217 2018-01-13 07:26:43,236 DEBUG [RS:1;asf911:49237] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:43,238 INFO [RS-EventLoopGroup-32-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.14 (auth:SIMPLE) 2018-01-13 07:26:43,238 INFO [RS-EventLoopGroup-32-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.15 (auth:SIMPLE) 2018-01-13 07:26:43,238 INFO [RS-EventLoopGroup-32-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 42030 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:43,239 INFO [RS-EventLoopGroup-32-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 41087 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:43,241 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,244 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=44597] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,245 DEBUG [RS:0;asf911:36099] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:43,245 DEBUG [RS:0;asf911:36099] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:43,245 DEBUG [RS:1;asf911:49237] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:43,245 DEBUG [RS:0;asf911:36099] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:43,245 DEBUG [RS:1;asf911:49237] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:43,245 DEBUG [RS:1;asf911:49237] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:43,275 INFO [M:0;asf911:44597] master.ServerManager(891): Finished wait on RegionServer count=2; waited=200ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:43,280 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:43,281 DEBUG [RS:0;asf911:36099] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,281 DEBUG [RS:1;asf911:49237] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,281 WARN [RS:0;asf911:36099] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:43,281 INFO [M:0;asf911:44597] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:43,281 WARN [RS:1;asf911:49237] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:43,281 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,281 DEBUG [RS:1;asf911:49237] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,281 DEBUG [RS:0;asf911:36099] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,282 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,290 INFO [RS:1;asf911:49237] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:43,291 INFO [RS:0;asf911:36099] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:43,292 DEBUG [RS:1;asf911:49237] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,293 DEBUG [RS:1;asf911:49237] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,293 DEBUG [RS:0;asf911:36099] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,293 DEBUG [RS:0;asf911:36099] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,294 INFO [RS:1;asf911:49237] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2bbebb5b to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:43,295 INFO [RS:0;asf911:36099] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x4d973fba to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:43,316 DEBUG [RS:1;asf911:49237] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18c6e89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:43,316 DEBUG [RS:1;asf911:49237] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:43,316 INFO [RS:1;asf911:49237] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:43,316 DEBUG [RS:0;asf911:36099] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@652c6ec5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:43,317 DEBUG [RS:0;asf911:36099] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:43,317 INFO [RS:0;asf911:36099] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:43,320 INFO [RS:1;asf911:49237] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:43,321 INFO [RS:0;asf911:36099] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:43,321 INFO [RS:1;asf911:49237] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:43,321 INFO [RS:0;asf911:36099] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:43,321 INFO [RS:1;asf911:49237] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:43,324 INFO [RS:0;asf911:36099] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:43,332 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:49237, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,332 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:36099, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,332 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:49237, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:43,332 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:36099, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:43,332 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:49237, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,332 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:36099, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,333 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:49237, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,333 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:36099, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,333 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:49237, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:43,333 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:36099, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:43,333 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:49237, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:43,334 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:36099, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:43,334 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:49237, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:43,334 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:36099, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:43,334 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:49237, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,334 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:36099, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:43,335 DEBUG [RS:1;asf911:49237] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:49237, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:43,335 DEBUG [RS:0;asf911:36099] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:36099, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:43,360 INFO [RS:1;asf911:49237] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:43,360 INFO [RS:0;asf911:36099] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:43,360 INFO [SplitLogWorker-asf911:36099] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,36099,1515828402164 starting 2018-01-13 07:26:43,360 INFO [SplitLogWorker-asf911:49237] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,49237,1515828402217 starting 2018-01-13 07:26:43,381 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,49237,1515828402217, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:49237, sessionid=0x160ee6894500069 2018-01-13 07:26:43,381 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:43,381 DEBUG [RS:1;asf911:49237] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,381 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,49237,1515828402217' 2018-01-13 07:26:43,381 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:43,381 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,36099,1515828402164, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:36099, sessionid=0x160ee6894500068 2018-01-13 07:26:43,382 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:43,384 DEBUG [RS:0;asf911:36099] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,385 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,36099,1515828402164' 2018-01-13 07:26:43,385 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:43,384 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:43,386 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:43,386 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:43,386 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:43,386 DEBUG [RS:1;asf911:49237] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:43,386 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,49237,1515828402217' 2018-01-13 07:26:43,386 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,36099,1515828402164' 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:43,387 DEBUG [RS:1;asf911:49237] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:43,387 DEBUG [RS:0;asf911:36099] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:43,387 DEBUG [RS:1;asf911:49237] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:43,387 INFO [RS:1;asf911:49237] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:43,387 INFO [RS:1;asf911:49237] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:43,388 DEBUG [RS:0;asf911:36099] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:43,388 INFO [RS:0;asf911:36099] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:43,388 INFO [RS:0;asf911:36099] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:43,501 DEBUG [M:0;asf911:44597] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:43,504 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:43,504 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:43,505 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:43,507 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:26:43,613 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:43,614 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:43,614 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:26:43,614 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:26:43,671 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:43,671 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:26:43,799 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x0fd6ca80 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:43,799 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(315): 0x6cd9acf6 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-13 07:26:43,822 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:26:43,822 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:26:43,833 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:43,834 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741896_1074, blk_1073741897_1075, blk_1073741898_1076, blk_1073741899_1077, blk_1073741900_1078, blk_1073741901_1079, blk_1073741902_1080, blk_1073741903_1081, blk_1073741904_1082, blk_1073741905_1083, blk_1073741906_1084, blk_1073741907_1085, blk_1073741908_1086] 2018-01-13 07:26:44,396 INFO [RS:1;asf911:49237] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C49237%2C1515828402217, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:44,396 INFO [RS:0;asf911:36099] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C36099%2C1515828402164, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:44,409 DEBUG [RS-EventLoopGroup-35-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:44,409 DEBUG [RS-EventLoopGroup-35-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:44,409 DEBUG [RS-EventLoopGroup-35-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:44,410 DEBUG [RS-EventLoopGroup-35-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:44,412 INFO [RS:0;asf911:36099] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 2018-01-13 07:26:44,412 DEBUG [RS:0;asf911:36099] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:44,413 INFO [RS:1;asf911:49237] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217/asf911.gq1.ygridcore.net%2C49237%2C1515828402217.1515828404397 2018-01-13 07:26:44,413 DEBUG [RS:1;asf911:49237] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:44,415 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,429 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:26:44,458 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,610 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,36099,1515828402164 version=3145728 2018-01-13 07:26:44,611 DEBUG [RSProcedureDispatcher-pool74-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,612 DEBUG [RSProcedureDispatcher-pool74-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:44,614 INFO [RS-EventLoopGroup-34-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:44,614 INFO [RS-EventLoopGroup-34-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40555 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:44,616 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36099] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:44,617 INFO [RS_OPEN_META-asf911:36099-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:44,621 INFO [RS_OPEN_META-asf911:36099-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:44,627 DEBUG [RS-EventLoopGroup-35-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:44,627 DEBUG [RS-EventLoopGroup-35-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:44,630 INFO [RS_OPEN_META-asf911:36099-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:44,630 DEBUG [RS_OPEN_META-asf911:36099-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:44,630 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:44,631 DEBUG [RS_OPEN_META-asf911:36099-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:44,631 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:44,631 INFO [RS_OPEN_META-asf911:36099-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:44,631 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:44,632 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:44,635 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:44,636 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:44,636 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:44,636 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:44,636 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:44,639 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:44,639 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:44,639 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:44,639 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:44,640 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:44,642 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:44,642 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:44,642 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:44,642 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:44,643 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:44,645 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:44,645 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:44,645 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:44,646 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:44,646 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:44,648 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:44,649 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:44,649 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:44,649 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:44,649 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:44,652 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:44,655 DEBUG [RS_OPEN_META-asf911:36099-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:44,659 DEBUG [RS_OPEN_META-asf911:36099-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:44,660 INFO [RS_OPEN_META-asf911:36099-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:26:44,664 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:44,670 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,671 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:44,672 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828404670}]},"ts":1515828404670} 2018-01-13 07:26:44,674 DEBUG [RS_OPEN_META-asf911:36099-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,679 DEBUG [hconnection-0x5df5234d-shared-pool77-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:44,681 INFO [RS-EventLoopGroup-34-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:44,682 INFO [RS-EventLoopGroup-34-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40558 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:44,685 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:44,687 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,698 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:44,761 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:26:44,807 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:44,807 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.0850sec 2018-01-13 07:26:44,848 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.5260sec 2018-01-13 07:26:44,849 INFO [M:0;asf911:44597] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:44,859 INFO [M:0;asf911:44597] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164 belongs to an existing region server 2018-01-13 07:26:44,860 INFO [M:0;asf911:44597] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217 belongs to an existing region server 2018-01-13 07:26:44,860 INFO [M:0;asf911:44597] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:44,862 INFO [M:0;asf911:44597] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:44,862 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828409862 2018-01-13 07:26:44,862 INFO [M:0;asf911:44597] assignment.AssignmentManager(1219): Joined the cluster in 2msec, failover=true 2018-01-13 07:26:44,867 INFO [M:0;asf911:44597] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:26:44,867 INFO [M:0;asf911:44597] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:26:45,040 DEBUG [M:0;asf911:44597] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:45,043 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:45,090 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:45,126 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741917_1095{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 476 2018-01-13 07:26:45,126 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741917_1095 size 476 2018-01-13 07:26:45,529 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:45,531 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:26:45,548 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741918_1096{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 42 2018-01-13 07:26:45,548 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741918_1096 size 42 2018-01-13 07:26:45,621 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:45,950 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:45,950 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.: disabling compactions & flushes 2018-01-13 07:26:45,950 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:45,950 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,086 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:46,088 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:46,093 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:46,139 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:46,140 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828406139}]},"ts":1515828406139} 2018-01-13 07:26:46,144 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:26:46,144 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,152 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217}] 2018-01-13 07:26:46,190 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217 hbase:namespace hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,192 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OFFLINE, location=asf911.gq1.ygridcore.net,49237,1515828402217; forceNewPlan=false, retain=false 2018-01-13 07:26:46,345 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:46,350 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,351 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2352ed20 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:46,373 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d95173a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:46,375 DEBUG [MultiHConnection-shared--pool78-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:46,377 INFO [RS-EventLoopGroup-34-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:46,378 INFO [RS-EventLoopGroup-34-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40580 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:46,381 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,534 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,49237,1515828402217 version=3145728 2018-01-13 07:26:46,535 DEBUG [RSProcedureDispatcher-pool74-t2] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,535 DEBUG [RSProcedureDispatcher-pool74-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:49237 2018-01-13 07:26:46,538 INFO [RS-EventLoopGroup-35-20] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:46,538 INFO [RS-EventLoopGroup-35-20] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40722 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:46,540 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=49237] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,546 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] regionserver.HRegion(6949): Opening region: {ENCODED => 59e7f350fc3769c7c320345d67c21835, NAME => 'hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:46,547 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 59e7f350fc3769c7c320345d67c21835 2018-01-13 07:26:46,547 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,552 DEBUG [StoreOpener-59e7f350fc3769c7c320345d67c21835-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/59e7f350fc3769c7c320345d67c21835/info 2018-01-13 07:26:46,552 DEBUG [StoreOpener-59e7f350fc3769c7c320345d67c21835-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/59e7f350fc3769c7c320345d67c21835/info 2018-01-13 07:26:46,552 INFO [StoreOpener-59e7f350fc3769c7c320345d67c21835-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:46,553 INFO [StoreOpener-59e7f350fc3769c7c320345d67c21835-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:46,553 INFO [StoreOpener-59e7f350fc3769c7c320345d67c21835-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:46,555 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/59e7f350fc3769c7c320345d67c21835 2018-01-13 07:26:46,561 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/59e7f350fc3769c7c320345d67c21835/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:46,562 INFO [RS_OPEN_PRIORITY_REGION-asf911:49237-0] regionserver.HRegion(960): Onlined 59e7f350fc3769c7c320345d67c21835; next sequenceid=2 2018-01-13 07:26:46,571 INFO [PostOpenDeployTasks:59e7f350fc3769c7c320345d67c21835] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,578 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,578 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,579 DEBUG [PostOpenDeployTasks:59e7f350fc3769c7c320345d67c21835] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:46,581 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:49237-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. on asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,631 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:46,678 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:26:46,679 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:46,679 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=59e7f350fc3769c7c320345d67c21835, target=asf911.gq1.ygridcore.net,49237,1515828402217 in 437msec 2018-01-13 07:26:46,679 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828406679}]},"ts":1515828406679} 2018-01-13 07:26:46,683 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:26:46,760 DEBUG [M:0;asf911:44597] zookeeper.ZKUtil(357): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:26:46,761 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:46,774 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:46,780 DEBUG [M:0;asf911:44597] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:49237 2018-01-13 07:26:46,782 INFO [RS-EventLoopGroup-35-22] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:46,783 INFO [RS-EventLoopGroup-35-22] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40724 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:46,813 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.8940sec 2018-01-13 07:26:46,834 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741896_1074, blk_1073741897_1075, blk_1073741898_1076, blk_1073741899_1077, blk_1073741900_1078, blk_1073741901_1079, blk_1073741902_1080, blk_1073741903_1081, blk_1073741904_1082, blk_1073741905_1083, blk_1073741906_1084, blk_1073741907_1085, blk_1073741908_1086] 2018-01-13 07:26:46,855 DEBUG [M:0;asf911:44597] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:46,857 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:46,896 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:46,938 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:46,988 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:47,004 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:47,059 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:47,101 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 275msec 2018-01-13 07:26:47,238 DEBUG [M:0;asf911:44597] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,241 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,271 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,305 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,338 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,354 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:47,396 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:47,430 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 286msec 2018-01-13 07:26:47,448 DEBUG [M:0;asf911:44597] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:47,457 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:47,465 DEBUG [M:0;asf911:44597] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:47,473 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:47,474 INFO [M:0;asf911:44597] master.HMaster(928): Master has completed initialization 5.160sec 2018-01-13 07:26:47,474 INFO [M:0;asf911:44597] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:47,475 INFO [M:0;asf911:44597] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:47,476 INFO [M:0;asf911:44597] regionserver.HRegionServer(866): ClusterId : e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:47,558 DEBUG [hconnection-0x51a18b21-shared-pool12-t8] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:47,560 DEBUG [Default-IPC-NioEventLoopGroup-7-25] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:47183 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:47183 2018-01-13 07:26:47,667 DEBUG [hconnection-0x51a18b21-shared-pool12-t8] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:47,670 INFO [RS-EventLoopGroup-34-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:47,670 INFO [RS-EventLoopGroup-34-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40592 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:47,673 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:47,700 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:47,700 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:47,701 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:47,701 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:47,701 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:47,701 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:47,704 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:59697 2018-01-13 07:26:47,704 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:47,705 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:47,710 INFO [Time-limited test] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:47,712 INFO [Time-limited test] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:47,714 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:59697 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:47,724 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:596970x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:47,725 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(523): regionserver:59697-0x160ee6894500071 connected 2018-01-13 07:26:47,725 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:47,726 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-01-13 07:26:47,727 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=59697 2018-01-13 07:26:47,728 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=59697 2018-01-13 07:26:47,729 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=59697 2018-01-13 07:26:47,733 INFO [RS:2;asf911:59697] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x6f556bba to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:47,747 DEBUG [RS:2;asf911:59697] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61929820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:47,750 INFO [RS:2;asf911:59697] regionserver.HRegionServer(866): ClusterId : e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:47,750 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:47,757 DEBUG [RS:2;asf911:59697] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:47,758 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:47,758 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:47,765 DEBUG [RS:2;asf911:59697] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:47,766 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:47,766 DEBUG [RS:2;asf911:59697] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65ad00ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:47,766 DEBUG [RS:2;asf911:59697] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:2;asf911:59697 2018-01-13 07:26:47,766 INFO [RS:2;asf911:59697] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:47,767 INFO [RS:2;asf911:59697] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:47,767 INFO [RS:2;asf911:59697] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,44597,1515828401985 with port=59697, startcode=1515828407699 2018-01-13 07:26:47,768 DEBUG [RS:2;asf911:59697] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:47,770 INFO [RS-EventLoopGroup-32-4] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.16 (auth:SIMPLE) 2018-01-13 07:26:47,770 INFO [RS-EventLoopGroup-32-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 48365 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:47,771 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,772 DEBUG [RS:2;asf911:59697] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:47,772 DEBUG [RS:2;asf911:59697] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:47,772 DEBUG [RS:2;asf911:59697] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:47,782 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:47,782 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:47,782 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:47,782 DEBUG [RS:2;asf911:59697] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,783 WARN [RS:2;asf911:59697] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:47,783 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:47,783 DEBUG [RS:2;asf911:59697] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,783 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:47,783 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:47,783 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:47,784 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:47,784 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,784 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:47,784 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,788 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,793 INFO [RS:2;asf911:59697] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:47,794 DEBUG [RS:2;asf911:59697] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:47,795 DEBUG [RS:2;asf911:59697] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:47,795 DEBUG [RS:2;asf911:59697] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,796 INFO [RS:2;asf911:59697] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x1c630b89 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:47,808 DEBUG [RS:2;asf911:59697] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40806d54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:47,808 DEBUG [RS:2;asf911:59697] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:47,809 INFO [RS:2;asf911:59697] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:47,812 INFO [RS:2;asf911:59697] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:47,812 INFO [RS:2;asf911:59697] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:47,813 INFO [RS:2;asf911:59697] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:47,818 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:59697, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:47,818 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:59697, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:47,818 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:59697, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:47,818 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:59697, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:47,818 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:59697, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:47,819 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:59697, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:47,819 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:59697, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:47,819 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:59697, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:47,819 DEBUG [RS:2;asf911:59697] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:59697, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:47,847 INFO [RS:2;asf911:59697] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:47,847 INFO [SplitLogWorker-asf911:59697] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,59697,1515828407699 starting 2018-01-13 07:26:47,873 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,59697,1515828407699, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:59697, sessionid=0x160ee6894500071 2018-01-13 07:26:47,874 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:47,874 DEBUG [RS:2;asf911:59697] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,874 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,59697,1515828407699' 2018-01-13 07:26:47,874 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:47,874 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:42935 2018-01-13 07:26:47,875 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:47,875 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:47,876 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:47,876 DEBUG [RS:2;asf911:59697] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:47,876 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,59697,1515828407699' 2018-01-13 07:26:47,876 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:47,876 DEBUG [RS:2;asf911:59697] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:47,877 DEBUG [RS:2;asf911:59697] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:47,877 INFO [RS:2;asf911:59697] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:47,877 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:42935 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42935 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42935 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:26:47,877 DEBUG [Default-IPC-NioEventLoopGroup-7-27] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:42935 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:42935 2018-01-13 07:26:47,877 INFO [RS:2;asf911:59697] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:47,884 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:47,887 INFO [RS-EventLoopGroup-32-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:47,887 INFO [RS-EventLoopGroup-32-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 52455 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:47,894 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.HMaster$3(1743): Client=jenkins//67.195.81.155 create 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-13 07:26:48,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,058 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:48,103 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,130 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741919_1097{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 517 2018-01-13 07:26:48,131 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741919_1097 size 517 2018-01-13 07:26:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:48,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:48,534 DEBUG [ProcExecWrkr-14] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:48,537 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,537 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,537 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,538 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,538 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,538 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(6772): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:48,623 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741920_1098{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:48,625 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741921_1099{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 84 2018-01-13 07:26:48,625 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:48,626 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.: disabling compactions & flushes 2018-01-13 07:26:48,626 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:48,626 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:48,626 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741922_1100{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 84 2018-01-13 07:26:48,626 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741921_1099 size 84 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741924_1102{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 85 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741923_1101{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741922_1100 size 84 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741920_1098 size 85 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741923_1101{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:48,627 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741924_1102 size 85 2018-01-13 07:26:48,628 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741925_1103{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:48,628 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741925_1103{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:48,629 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:48,629 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.: disabling compactions & flushes 2018-01-13 07:26:48,629 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:48,629 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:48,629 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:48,629 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.: disabling compactions & flushes 2018-01-13 07:26:48,630 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:48,630 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:48,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:48,888 INFO [RS:2;asf911:59697] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C59697%2C1515828407699, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:48,898 DEBUG [RS-EventLoopGroup-36-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:48,898 DEBUG [RS-EventLoopGroup-36-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:48,902 INFO [RS:2;asf911:59697] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699/asf911.gq1.ygridcore.net%2C59697%2C1515828407699.1515828408888 2018-01-13 07:26:48,902 DEBUG [RS:2;asf911:59697] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:49,026 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,026 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,026 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.: disabling compactions & flushes 2018-01-13 07:26:49,026 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.: disabling compactions & flushes 2018-01-13 07:26:49,027 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,027 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,027 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.: disabling compactions & flushes 2018-01-13 07:26:49,027 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,027 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,027 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,027 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,027 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,078 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:49,080 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,080 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,081 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,081 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,081 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.","families":{"info":[{"qualifier":"regioninfo","vlen":83,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,081 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.","families":{"info":[{"qualifier":"regioninfo","vlen":84,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:49,087 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1638): Added 6 regions to meta. 2018-01-13 07:26:49,155 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:49,156 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828409156}]},"ts":1515828409156} 2018-01-13 07:26:49,160 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1786): Updated table testLogSplittingAfterMasterRecoveryDueToZKExpiry state to ENABLING in META 2018-01-13 07:26:49,160 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,166 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,167 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,168 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:49,169 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,170 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,171 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217}, {pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164}, {pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217}, {pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699}, {pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699}, {pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164}] 2018-01-13 07:26:49,213 INFO [ProcExecWrkr-16] procedure.MasterProcedureScheduler(883): pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,214 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,214 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(883): pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,215 INFO [ProcExecWrkr-15] procedure.MasterProcedureScheduler(883): pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,215 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,215 INFO [ProcExecWrkr-4] procedure.MasterProcedureScheduler(883): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,216 INFO [ProcExecWrkr-16] assignment.AssignProcedure(211): Start pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59697,1515828407699; forceNewPlan=false, retain=false 2018-01-13 07:26:49,216 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OFFLINE, location=asf911.gq1.ygridcore.net,49237,1515828402217; forceNewPlan=false, retain=false 2018-01-13 07:26:49,216 INFO [ProcExecWrkr-3] assignment.AssignProcedure(211): Start pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164; forceNewPlan=false, retain=false 2018-01-13 07:26:49,216 INFO [ProcExecWrkr-15] assignment.AssignProcedure(211): Start pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164; forceNewPlan=false, retain=false 2018-01-13 07:26:49,216 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59697,1515828407699; forceNewPlan=false, retain=false 2018-01-13 07:26:49,217 INFO [ProcExecWrkr-4] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OFFLINE, location=asf911.gq1.ygridcore.net,49237,1515828402217; forceNewPlan=false, retain=false 2018-01-13 07:26:49,319 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:49,320 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:49,370 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 6 regions. 6 retained the pre-restart assignment. 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-5] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-8] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-7] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,381 INFO [ProcExecWrkr-6] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,388 INFO [ProcExecWrkr-5] assignment.RegionTransitionProcedure(208): Dispatch pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,388 INFO [ProcExecWrkr-8] assignment.RegionTransitionProcedure(208): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,389 INFO [ProcExecWrkr-7] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,388 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,388 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(208): Dispatch pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,389 INFO [ProcExecWrkr-6] assignment.RegionTransitionProcedure(208): Dispatch pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,539 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,59697,1515828407699 version=3145728 2018-01-13 07:26:49,539 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,36099,1515828402164 version=3145728 2018-01-13 07:26:49,540 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,49237,1515828402217 version=3145728 2018-01-13 07:26:49,540 DEBUG [RSProcedureDispatcher-pool74-t3] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,550 DEBUG [RSProcedureDispatcher-pool74-t3] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:26:49,550 INFO [RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=36099] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,550 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=49237] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,552 INFO [RS-EventLoopGroup-36-7] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:49,552 INFO [RS-EventLoopGroup-36-7] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34163 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:49,555 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59697] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,558 INFO [RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=36099] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,558 DEBUG [RS_OPEN_REGION-asf911:36099-0] regionserver.HRegion(6949): Opening region: {ENCODED => 2436b98bf1d1a818c8c836b539d7c151, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.', STARTKEY => '5', ENDKEY => ''} 2018-01-13 07:26:49,559 DEBUG [RS_OPEN_REGION-asf911:36099-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2436b98bf1d1a818c8c836b539d7c151 2018-01-13 07:26:49,559 DEBUG [RS_OPEN_REGION-asf911:36099-1] regionserver.HRegion(6949): Opening region: {ENCODED => b4402e59bcc9a4f34ad3070698f03ed5, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.', STARTKEY => '1', ENDKEY => '2'} 2018-01-13 07:26:49,559 DEBUG [RS_OPEN_REGION-asf911:36099-0] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,559 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=49237] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,559 DEBUG [RS_OPEN_REGION-asf911:36099-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry b4402e59bcc9a4f34ad3070698f03ed5 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:49237-0] regionserver.HRegion(6949): Opening region: {ENCODED => 02aa6cdb76f8c3ab882fdc1614996b03, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.', STARTKEY => '', ENDKEY => '1'} 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:36099-1] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:49237-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 02aa6cdb76f8c3ab882fdc1614996b03 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:49237-0] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:49237-1] regionserver.HRegion(6949): Opening region: {ENCODED => b15455530ee6b30edbc7069733984b38, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.', STARTKEY => '2', ENDKEY => '3'} 2018-01-13 07:26:49,560 DEBUG [RS_OPEN_REGION-asf911:59697-0] regionserver.HRegion(6949): Opening region: {ENCODED => 77233b6b05a0a35d88c2b7315948add1, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.', STARTKEY => '3', ENDKEY => '4'} 2018-01-13 07:26:49,560 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59697] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,561 DEBUG [RS_OPEN_REGION-asf911:59697-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 77233b6b05a0a35d88c2b7315948add1 2018-01-13 07:26:49,561 DEBUG [RS_OPEN_REGION-asf911:49237-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry b15455530ee6b30edbc7069733984b38 2018-01-13 07:26:49,561 DEBUG [RS_OPEN_REGION-asf911:59697-0] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,561 DEBUG [RS_OPEN_REGION-asf911:49237-1] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,562 DEBUG [RS_OPEN_REGION-asf911:59697-1] regionserver.HRegion(6949): Opening region: {ENCODED => aed90a6304ebc698cae059a0ac39bff6, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.', STARTKEY => '4', ENDKEY => '5'} 2018-01-13 07:26:49,562 DEBUG [RS_OPEN_REGION-asf911:59697-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry aed90a6304ebc698cae059a0ac39bff6 2018-01-13 07:26:49,562 DEBUG [RS_OPEN_REGION-asf911:59697-1] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,567 DEBUG [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/col 2018-01-13 07:26:49,567 DEBUG [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/col 2018-01-13 07:26:49,567 DEBUG [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02aa6cdb76f8c3ab882fdc1614996b03/col 2018-01-13 07:26:49,567 DEBUG [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02aa6cdb76f8c3ab882fdc1614996b03/col 2018-01-13 07:26:49,568 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,568 DEBUG [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/col 2018-01-13 07:26:49,568 DEBUG [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/col 2018-01-13 07:26:49,568 INFO [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,568 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,568 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,568 DEBUG [StoreOpener-b15455530ee6b30edbc7069733984b38-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b15455530ee6b30edbc7069733984b38/col 2018-01-13 07:26:49,569 DEBUG [StoreOpener-b15455530ee6b30edbc7069733984b38-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b15455530ee6b30edbc7069733984b38/col 2018-01-13 07:26:49,568 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,569 INFO [StoreOpener-b15455530ee6b30edbc7069733984b38-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,569 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,569 DEBUG [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/77233b6b05a0a35d88c2b7315948add1/col 2018-01-13 07:26:49,569 DEBUG [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/77233b6b05a0a35d88c2b7315948add1/col 2018-01-13 07:26:49,569 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,568 INFO [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,570 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,570 INFO [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,570 INFO [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,569 DEBUG [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/aed90a6304ebc698cae059a0ac39bff6/col 2018-01-13 07:26:49,571 DEBUG [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/aed90a6304ebc698cae059a0ac39bff6/col 2018-01-13 07:26:49,569 INFO [StoreOpener-b15455530ee6b30edbc7069733984b38-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,571 INFO [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:49,571 INFO [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:49,571 INFO [RS_OPEN_REGION-asf911:36099-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,570 INFO [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,570 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,570 INFO [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,572 INFO [StoreOpener-77233b6b05a0a35d88c2b7315948add1-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,572 INFO [StoreOpener-02aa6cdb76f8c3ab882fdc1614996b03-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,571 INFO [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,571 INFO [StoreOpener-b15455530ee6b30edbc7069733984b38-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:49,572 DEBUG [RS_OPEN_REGION-asf911:36099-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151 2018-01-13 07:26:49,572 INFO [StoreOpener-aed90a6304ebc698cae059a0ac39bff6-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,573 INFO [RS_OPEN_REGION-asf911:59697-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,573 INFO [RS_OPEN_REGION-asf911:49237-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,572 INFO [RS_OPEN_REGION-asf911:36099-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,573 INFO [StoreOpener-b15455530ee6b30edbc7069733984b38-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:49,574 DEBUG [RS_OPEN_REGION-asf911:49237-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02aa6cdb76f8c3ab882fdc1614996b03 2018-01-13 07:26:49,574 DEBUG [RS_OPEN_REGION-asf911:36099-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5 2018-01-13 07:26:49,574 INFO [RS_OPEN_REGION-asf911:59697-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,574 DEBUG [RS_OPEN_REGION-asf911:59697-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/77233b6b05a0a35d88c2b7315948add1 2018-01-13 07:26:49,574 INFO [RS_OPEN_REGION-asf911:49237-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,575 DEBUG [RS_OPEN_REGION-asf911:59697-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/aed90a6304ebc698cae059a0ac39bff6 2018-01-13 07:26:49,575 DEBUG [RS_OPEN_REGION-asf911:36099-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,575 DEBUG [RS_OPEN_REGION-asf911:49237-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b15455530ee6b30edbc7069733984b38 2018-01-13 07:26:49,576 DEBUG [RS_OPEN_REGION-asf911:49237-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,576 DEBUG [RS_OPEN_REGION-asf911:36099-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,577 DEBUG [RS_OPEN_REGION-asf911:59697-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,578 DEBUG [RS_OPEN_REGION-asf911:59697-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,578 DEBUG [RS_OPEN_REGION-asf911:49237-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:49,583 DEBUG [RS_OPEN_REGION-asf911:36099-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,583 DEBUG [RS_OPEN_REGION-asf911:49237-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/02aa6cdb76f8c3ab882fdc1614996b03/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,583 INFO [RS_OPEN_REGION-asf911:49237-0] regionserver.HRegion(960): Onlined 02aa6cdb76f8c3ab882fdc1614996b03; next sequenceid=2 2018-01-13 07:26:49,583 INFO [RS_OPEN_REGION-asf911:36099-0] regionserver.HRegion(960): Onlined 2436b98bf1d1a818c8c836b539d7c151; next sequenceid=2 2018-01-13 07:26:49,588 DEBUG [RS_OPEN_REGION-asf911:36099-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,588 INFO [RS_OPEN_REGION-asf911:36099-1] regionserver.HRegion(960): Onlined b4402e59bcc9a4f34ad3070698f03ed5; next sequenceid=2 2018-01-13 07:26:49,589 DEBUG [RS_OPEN_REGION-asf911:59697-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/77233b6b05a0a35d88c2b7315948add1/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,589 INFO [RS_OPEN_REGION-asf911:59697-0] regionserver.HRegion(960): Onlined 77233b6b05a0a35d88c2b7315948add1; next sequenceid=2 2018-01-13 07:26:49,589 DEBUG [RS_OPEN_REGION-asf911:49237-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b15455530ee6b30edbc7069733984b38/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,590 INFO [PostOpenDeployTasks:02aa6cdb76f8c3ab882fdc1614996b03] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,590 INFO [RS_OPEN_REGION-asf911:49237-1] regionserver.HRegion(960): Onlined b15455530ee6b30edbc7069733984b38; next sequenceid=2 2018-01-13 07:26:49,590 INFO [PostOpenDeployTasks:b4402e59bcc9a4f34ad3070698f03ed5] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,593 DEBUG [RS_OPEN_REGION-asf911:59697-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/aed90a6304ebc698cae059a0ac39bff6/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:49,593 INFO [RS_OPEN_REGION-asf911:59697-1] regionserver.HRegion(960): Onlined aed90a6304ebc698cae059a0ac39bff6; next sequenceid=2 2018-01-13 07:26:49,593 INFO [PostOpenDeployTasks:2436b98bf1d1a818c8c836b539d7c151] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,596 INFO [PostOpenDeployTasks:b15455530ee6b30edbc7069733984b38] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,599 DEBUG [PostOpenDeployTasks:02aa6cdb76f8c3ab882fdc1614996b03] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:49,599 INFO [ProcExecWrkr-11] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,603 INFO [PostOpenDeployTasks:aed90a6304ebc698cae059a0ac39bff6] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,601 INFO [PostOpenDeployTasks:77233b6b05a0a35d88c2b7315948add1] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,606 INFO [ProcExecWrkr-13] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,607 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,606 INFO [ProcExecWrkr-12] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,606 DEBUG [PostOpenDeployTasks:b4402e59bcc9a4f34ad3070698f03ed5] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:49,603 DEBUG [RS_OPEN_REGION-asf911:49237-0] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. on asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44597] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,612 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,607 DEBUG [PostOpenDeployTasks:b15455530ee6b30edbc7069733984b38] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:49,606 DEBUG [PostOpenDeployTasks:2436b98bf1d1a818c8c836b539d7c151] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:49,616 DEBUG [RS_OPEN_REGION-asf911:49237-1] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. on asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,614 INFO [ProcExecWrkr-3] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,614 DEBUG [PostOpenDeployTasks:aed90a6304ebc698cae059a0ac39bff6] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:49,614 DEBUG [PostOpenDeployTasks:77233b6b05a0a35d88c2b7315948add1] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:49,614 DEBUG [RS_OPEN_REGION-asf911:36099-1] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. on asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,622 DEBUG [RS_OPEN_REGION-asf911:59697-1] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. on asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,619 DEBUG [RS_OPEN_REGION-asf911:36099-0] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. on asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,625 DEBUG [RS_OPEN_REGION-asf911:59697-0] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. on asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=9, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=9, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5, target=asf911.gq1.ygridcore.net,36099,1515828402164 in 456msec 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=13, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b15455530ee6b30edbc7069733984b38, target=asf911.gq1.ygridcore.net,49237,1515828402217 in 455msec 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=02aa6cdb76f8c3ab882fdc1614996b03, target=asf911.gq1.ygridcore.net,49237,1515828402217 in 454msec 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=12, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,706 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=11, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:49,707 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1250): Finished pid=12, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=aed90a6304ebc698cae059a0ac39bff6, target=asf911.gq1.ygridcore.net,59697,1515828407699 in 457msec 2018-01-13 07:26:49,707 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=13, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151, target=asf911.gq1.ygridcore.net,36099,1515828402164 in 455msec 2018-01-13 07:26:49,753 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry; resume parent processing. 2018-01-13 07:26:49,753 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:49,753 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=11, ppid=7, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=77233b6b05a0a35d88c2b7315948add1, target=asf911.gq1.ygridcore.net,59697,1515828407699 in 456msec 2018-01-13 07:26:49,754 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testLogSplittingAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828409753}]},"ts":1515828409753} 2018-01-13 07:26:49,758 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1786): Updated table testLogSplittingAfterMasterRecoveryDueToZKExpiry state to ENABLED in META 2018-01-13 07:26:49,803 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:49,868 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry in 1.9090sec 2018-01-13 07:26:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44597] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:26:50,171 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3638): Operation: CREATE, Table Name: default:testLogSplittingAfterMasterRecoveryDueToZKExpiry completed 2018-01-13 07:26:50,175 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-13 07:26:50,179 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:49237 2018-01-13 07:26:50,183 INFO [RS-EventLoopGroup-35-24] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:50,183 INFO [RS-EventLoopGroup-35-24] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40775 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:50,215 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,215 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,215 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,215 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:50,216 ERROR [Time-limited test] helpers.MarkerIgnoringBase(159): Test recovery from zk session expired org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:434) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,216 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:50,215 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,215 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,216 WARN [main-EventThread] zookeeper.ZKUtil(361): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,217 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,216 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:50,216 INFO [Time-limited test] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,44597,1515828401985' ***** 2018-01-13 07:26:50,217 INFO [Time-limited test] regionserver.HRegionServer(2172): STOPPED: Stopped by Time-limited test 2018-01-13 07:26:50,216 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:50,218 WARN [Time-limited test] zookeeper.ZKUtil(618): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:434) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,218 ERROR [Time-limited test] zookeeper.ZKWatcher(578): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:434) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,217 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:50,218 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): Unexpected exception handling nodeDeleted event org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,217 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:50,216 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:50,219 DEBUG [main-EventThread] master.ActiveMasterManager(132): No master available. Notifying waiting threads 2018-01-13 07:26:50,218 WARN [main-EventThread] zookeeper.ZKUtil(618): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,218 ERROR [Time-limited test] master.ActiveMasterManager(280): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testLogSplittingAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:434) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,218 DEBUG [M:0;asf911:44597] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@276e6e30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:50,220 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,36099,1515828402164: Aborting ***** 2018-01-13 07:26:50,220 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:50,219 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,220 ERROR [main-EventThread] master.ActiveMasterManager(280): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:26:50,220 INFO [M:0;asf911:44597] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,44597,1515828401985 2018-01-13 07:26:50,221 DEBUG [M:0;asf911:44597] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:50,224 INFO [M:0;asf911:44597] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x08855668 to localhost:59094 2018-01-13 07:26:50,226 DEBUG [M:0;asf911:44597] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:50,227 INFO [M:0;asf911:44597] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,44597,1515828401985; all regions closed. 2018-01-13 07:26:50,228 DEBUG [M:0;asf911:44597] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:50,228 INFO [M:0;asf911:44597] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,44597,1515828401985 had [[ScheduledChore: Name: CatalogJanitor-asf911:44597 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,44597,1515828401985-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,44597,1515828401985-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,44597,1515828401985-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,44597,1515828401985-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,44597,1515828401985-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS]] on shutdown 2018-01-13 07:26:50,228 INFO [M:0;asf911:44597] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:50,228 INFO [M:0;asf911:44597] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:50,228 DEBUG [M:0;asf911:44597] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:50,228 INFO [Time-limited test] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 832729848 }, "NonHeapMemoryUsage" : { "committed" : 137035776, "init" : 2555904, "max" : -1, "used" : 134882864 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 57714, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 268, "TotalCallTime_min" : 0, "TotalCallTime_max" : 11, "TotalCallTime_mean" : 2, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 2, "TotalCallTime_90th_percentile" : 5, "TotalCallTime_95th_percentile" : 11, "TotalCallTime_98th_percentile" : 11, "TotalCallTime_99th_percentile" : 11, "TotalCallTime_99.9th_percentile" : 11, "TotalCallTime_TimeRangeCount_0-1" : 8, "TotalCallTime_TimeRangeCount_1-3" : 7, "TotalCallTime_TimeRangeCount_3-10" : 1, "TotalCallTime_TimeRangeCount_10-30" : 1, "TotalCallTime_TimeRangeCount_600000-inf" : 251, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 268, "ResponseSize_min" : 0, "ResponseSize_max" : 1006, "ResponseSize_mean" : 75, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 7, "ResponseSize_75th_percentile" : 21, "ResponseSize_90th_percentile" : 90, "ResponseSize_95th_percentile" : 898, "ResponseSize_98th_percentile" : 963, "ResponseSize_99th_percentile" : 984, "ResponseSize_99.9th_percentile" : 1003, "ResponseSize_SizeRangeCount_0-10" : 12, "ResponseSize_SizeRangeCount_10-100" : 4, "ResponseSize_SizeRangeCount_100-1000" : 1, "ResponseSize_SizeRangeCount_100000000-inf" : 251, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 268, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 11, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 2, "ProcessCallTime_90th_percentile" : 5, "ProcessCallTime_95th_percentile" : 11, "ProcessCallTime_98th_percentile" : 11, "ProcessCallTime_99th_percentile" : 11, "ProcessCallTime_99.9th_percentile" : 11, "ProcessCallTime_TimeRangeCount_0-1" : 8, "ProcessCallTime_TimeRangeCount_1-3" : 7, "ProcessCallTime_TimeRangeCount_3-10" : 1, "ProcessCallTime_TimeRangeCount_10-30" : 1, "ProcessCallTime_TimeRangeCount_600000-inf" : 251, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 44, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 268, "RequestSize_min" : 78, "RequestSize_max" : 978, "RequestSize_mean" : 229, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 121, "RequestSize_75th_percentile" : 196, "RequestSize_90th_percentile" : 734, "RequestSize_95th_percentile" : 856, "RequestSize_98th_percentile" : 929, "RequestSize_99th_percentile" : 953, "RequestSize_99.9th_percentile" : 975, "RequestSize_SizeRangeCount_10-100" : 6, "RequestSize_SizeRangeCount_100-1000" : 11, "RequestSize_SizeRangeCount_100000000-inf" : 251, "sentBytes" : 28218, "QueueCallTime_num_ops" : 268, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 17, "QueueCallTime_TimeRangeCount_600000-inf" : 251, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:50,229 WARN [M:0;asf911:44597] zookeeper.ZKUtil(618): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,231 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,231 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,229 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,231 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44597] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,36099,1515828402164 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,36099,1515828402164: Aborting ***** 2018-01-13 07:26:50,231 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:50,231 ERROR [M:0;asf911:44597] zookeeper.ZKWatcher(578): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,232 ERROR [M:0;asf911:44597] master.ActiveMasterManager(280): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,232 INFO [Time-limited test] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,36099,1515828402164' ***** 2018-01-13 07:26:50,232 INFO [Time-limited test] regionserver.HRegionServer(2172): STOPPED: Aborting 2018-01-13 07:26:50,232 INFO [M:0;asf911:44597] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:50,233 INFO [RS:0;asf911:36099] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:50,233 INFO [RS:0;asf911:36099] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:50,233 INFO [M:0;asf911:44597] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x2352ed20 to localhost:59094 2018-01-13 07:26:50,233 INFO [SplitLogWorker-asf911:36099] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:50,235 INFO [SplitLogWorker-asf911:36099] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,36099,1515828402164 exiting 2018-01-13 07:26:50,235 DEBUG [M:0;asf911:44597] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:50,233 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:50,235 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:50,233 INFO [RS:0;asf911:36099] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:50,236 INFO [RS:0;asf911:36099] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:50,236 INFO [M:0;asf911:44597] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:50,238 DEBUG [RS_CLOSE_REGION-asf911:36099-0] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:50,238 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,238 DEBUG [RS:0;asf911:36099] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:50,238 INFO [M:0;asf911:44597] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:50,238 DEBUG [RS_CLOSE_REGION-asf911:36099-1] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:50,246 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [RS_CLOSE_REGION-asf911:36099-1] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.: disabling compactions & flushes 2018-01-13 07:26:50,247 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:50,238 INFO [RS:0;asf911:36099] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x0db9783d to localhost:59094 2018-01-13 07:26:50,248 DEBUG [RS:0;asf911:36099] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:50,248 INFO [RS:0;asf911:36099] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:50,248 INFO [RS:0;asf911:36099] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at '1', inclusive 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [RS_CLOSE_REGION-asf911:36099-1] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,247 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,246 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,246 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,246 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,246 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:50,244 DEBUG [RS_CLOSE_REGION-asf911:36099-0] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.: disabling compactions & flushes 2018-01-13 07:26:50,238 INFO [M:0;asf911:44597] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:50,250 DEBUG [RS_CLOSE_REGION-asf911:36099-0] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:50,251 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=36099] ipc.CallRunner(141): callId: 118 service: ClientService methodName: Scan size: 140 connection: 67.195.81.155:40592 deadline: 1515828470250 org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,36099,1515828402164 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1415) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3152) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41548) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:50,250 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:50,248 INFO [RS:0;asf911:36099] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:50,251 INFO [RS_CLOSE_REGION-asf911:36099-1] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:50,251 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:50,251 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 07:26:50,252 INFO [RS_CLOSE_REGION-asf911:36099-0] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:50,251 DEBUG [RS_CLOSE_REGION-asf911:36099-1] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:50,252 DEBUG [RS_CLOSE_REGION-asf911:36099-0] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:50,252 DEBUG [RS:0;asf911:36099] regionserver.HRegionServer(1392): Online Regions={b4402e59bcc9a4f34ad3070698f03ed5=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5., 1588230740=hbase:meta,,1.1588230740} 2018-01-13 07:26:50,252 DEBUG [RS_CLOSE_META-asf911:36099-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:50,252 DEBUG [RS_CLOSE_META-asf911:36099-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:50,252 DEBUG [RS_CLOSE_META-asf911:36099-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:50,253 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:50,253 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:50,253 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:50,253 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741913_1091{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 29146 2018-01-13 07:26:50,253 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:50,254 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741913_1091 size 29146 2018-01-13 07:26:50,254 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:50,254 ERROR [RS_CLOSE_META-asf911:36099-0] regionserver.HRegion(1620): Memstore size is 11452 2018-01-13 07:26:50,254 DEBUG [RS_CLOSE_META-asf911:36099-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:50,256 INFO [RS_CLOSE_META-asf911:36099-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:50,256 DEBUG [RS_CLOSE_META-asf911:36099-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:50,257 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:50,257 INFO [M:1;asf911:40421] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,40421,1515828402111 from backup master directory 2018-01-13 07:26:50,257 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:50,273 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:50,273 WARN [M:1;asf911:40421] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:50,274 INFO [M:1;asf911:40421] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:50,287 DEBUG [M:1;asf911:40421] util.FSTableDescriptors(787): Current tableInfoPath = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:50,291 DEBUG [M:1;asf911:40421] util.FSTableDescriptors(791): TableInfo already exists.. Skipping creation 2018-01-13 07:26:50,303 INFO [M:1;asf911:40421] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:50,304 INFO [M:1;asf911:40421] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:50,305 DEBUG [M:1;asf911:40421] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:50,324 INFO [M:1;asf911:40421] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x367ec517 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:50,334 DEBUG [M:1;asf911:40421] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22e1b57d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:50,339 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:50,339 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:50,353 INFO [M:1;asf911:40421] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:50,353 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=36099] ipc.CallRunner(141): callId: 119 service: ClientService methodName: Scan size: 140 connection: 67.195.81.155:40592 deadline: 1515828470249 org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,36099,1515828402164 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1415) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3152) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41548) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:50,354 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:50,354 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:50,355 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:50,356 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:50,356 INFO [M:1;asf911:40421] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:50,357 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,358 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,359 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,361 INFO [M:1;asf911:40421] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,40421,1515828402111, sessionid=0x160ee6894500067, setting cluster-up flag (Was=true) 2018-01-13 07:26:50,371 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:50,372 INFO [M:1;asf911:40421] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:50,373 DEBUG [M:1;asf911:40421] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:50,382 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:50,383 INFO [M:1;asf911:40421] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:50,384 DEBUG [M:1;asf911:40421] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:50,386 INFO [M:1;asf911:40421] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:50,386 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:40421, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:50,386 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:40421, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:50,386 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:40421, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:50,386 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:40421, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:50,386 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:40421, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:50,387 DEBUG [M:1;asf911:40421] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:40421, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:50,388 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:50,388 INFO [M:1;asf911:40421] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:50,389 INFO [M:1;asf911:40421] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:26:50,390 WARN [IPC Server handler 8 on 45316] namenode.FSNamesystem(4004): BLOCK* internalReleaseLease: All existing blocks are COMPLETE, lease removed, file closed. 2018-01-13 07:26:50,391 INFO [M:1;asf911:40421] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log after 2ms 2018-01-13 07:26:50,391 DEBUG [M:1;asf911:40421] wal.WALProcedureStore(1286): Opening Pv2 FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=29146; replication=2; blocksize=134217728; modification_time=1515828403065; access_time=1515828403065; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-01-13 07:26:50,398 DEBUG [M:1;asf911:40421] wal.WALProcedureStore(1071): Roll new state log: 2 2018-01-13 07:26:50,399 INFO [M:1;asf911:40421] wal.WALProcedureStore(390): Lease acquired for flushLogId: 2 2018-01-13 07:26:50,399 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 11msec 2018-01-13 07:26:50,404 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor$1(317): Load max pid=13 2018-01-13 07:26:50,405 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(356): Completed pid=7, state=SUCCESS; CreateTableProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:26:50,406 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(356): Completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:26:50,406 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(356): Completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:50,406 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(356): Completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:50,407 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(356): Completed pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:50,407 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 8msec 2018-01-13 07:26:50,409 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828440409 2018-01-13 07:26:50,409 INFO [M:1;asf911:40421] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:50,409 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:50,410 INFO [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:50,410 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:50,410 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:50,410 INFO [M:1;asf911:40421] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:50,413 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:50,413 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:50,413 DEBUG [M:1;asf911:40421] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:50,413 DEBUG [M:1;asf911:40421] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:40421-HFileCleaner.large.0-1515828410413,5,main] 2018-01-13 07:26:50,414 DEBUG [M:1;asf911:40421] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:40421-HFileCleaner.small.0-1515828410414,5,main] 2018-01-13 07:26:50,414 INFO [M:1;asf911:40421] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828402282, completionTime=-1 2018-01-13 07:26:50,414 INFO [M:1;asf911:40421] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:50,421 DEBUG [M:1;asf911:40421-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:50,423 DEBUG [M:1;asf911:40421-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500075 connected 2018-01-13 07:26:50,428 INFO [asf911.gq1.ygridcore.net,36099,1515828402164_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 07:26:50,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(156): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:50,452 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,36099,1515828402164; all regions closed. 2018-01-13 07:26:50,455 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741916_1094{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:50,456 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741916_1094{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:26:50,462 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741914_1092{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 874 2018-01-13 07:26:50,462 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 not finished, retry = 0 2018-01-13 07:26:50,462 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741914_1092 size 874 2018-01-13 07:26:50,557 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=36099] ipc.CallRunner(141): callId: 120 service: ClientService methodName: Scan size: 140 connection: 67.195.81.155:40592 deadline: 1515828470250 org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,36099,1515828402164 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1415) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3152) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41548) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 07:26:50,563 DEBUG [RS:0;asf911:36099] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:50,564 INFO [RS:0;asf911:36099] regionserver.Leases(147): RS:0;asf911:36099 closing leases 2018-01-13 07:26:50,564 INFO [RS:0;asf911:36099] regionserver.Leases(150): RS:0;asf911:36099 closed leases 2018-01-13 07:26:50,564 INFO [RS:0;asf911:36099] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,36099,1515828402164 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,36099,1515828402164 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:50,564 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:50,565 INFO [RS:0;asf911:36099] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:50,573 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,573 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,573 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,573 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:50,573 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:36099-0x160ee6894500068, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:50,582 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,36099,1515828402164] 2018-01-13 07:26:50,582 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,36099,1515828402164; zookeeper connection closed. 2018-01-13 07:26:50,582 INFO [RS:0;asf911:36099] regionserver.HRegionServer(1157): RS:0;asf911:36099 exiting 2018-01-13 07:26:50,582 WARN [main-EventThread] master.RegionServerTracker(126): asf911.gq1.ygridcore.net,36099,1515828402164 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2018-01-13 07:26:50,582 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,582 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@21aeabe0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@21aeabe0 2018-01-13 07:26:50,582 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,582 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:50,582 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,585 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 znode expired, triggering replicatorRemoved event 2018-01-13 07:26:50,586 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,586 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:50,586 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,36099,1515828402164 znode expired, triggering replicatorRemoved event 2018-01-13 07:26:50,586 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:50,586 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,586 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,586 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,588 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,588 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:50,588 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:50,655 INFO [M:0;asf911:44597] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,44597,1515828401985_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:50,655 INFO [M:0;asf911:44597] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:50,655 INFO [M:0;asf911:44597] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:50,656 WARN [M:0;asf911:44597] regionserver.HRegionServer(1146): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf911.gq1.ygridcore.net,44597,1515828401985 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1591) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1143) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:26:50,660 INFO [M:0;asf911:44597] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,44597,1515828401985; zookeeper connection closed. 2018-01-13 07:26:50,660 INFO [M:0;asf911:44597] regionserver.HRegionServer(1157): M:0;asf911:44597 exiting 2018-01-13 07:26:50,861 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t5] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:50,862 DEBUG [Default-IPC-NioEventLoopGroup-7-30] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:36099 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:50,908 DEBUG [RS:2;asf911:59697] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:50,909 DEBUG [RS-EventLoopGroup-36-8] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:44597 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:51,371 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t5] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:36099 this server is in the failed servers list 2018-01-13 07:26:51,434 DEBUG [RS:1;asf911:49237] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:51,435 DEBUG [RS-EventLoopGroup-35-25] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:44597 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:26:51,911 DEBUG [RS:2;asf911:59697] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:40421 2018-01-13 07:26:51,913 INFO [RS-EventLoopGroup-33-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.16 (auth:SIMPLE) 2018-01-13 07:26:51,914 INFO [RS-EventLoopGroup-33-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 37302 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:51,917 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:51,918 INFO [M:1;asf911:40421] master.ServerManager(871): Waiting on RegionServer count=1 to settle; waited=1504ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:52,053 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:52,376 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t5] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:36099 this server is in the failed servers list 2018-01-13 07:26:52,437 DEBUG [RS:1;asf911:49237] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:40421 2018-01-13 07:26:52,439 INFO [RS-EventLoopGroup-33-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.15 (auth:SIMPLE) 2018-01-13 07:26:52,439 INFO [RS-EventLoopGroup-33-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 38810 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:52,440 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:52,470 INFO [M:1;asf911:40421] master.ServerManager(891): Finished wait on RegionServer count=2; waited=2056ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:52,474 INFO [M:1;asf911:40421] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:52,669 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(869): Stored pid=14, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:52,671 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=14, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:52,671 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=14, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:52,674 INFO [ProcExecWrkr-1] master.MasterWalManager(185): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164 doesn't belong to a known region server, splitting 2018-01-13 07:26:52,675 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217 belongs to an existing region server 2018-01-13 07:26:52,676 INFO [ProcExecWrkr-1] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699 belongs to an existing region server 2018-01-13 07:26:52,678 DEBUG [ProcExecWrkr-1] master.MasterWalManager(257): Renamed region directory: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting 2018-01-13 07:26:52,678 INFO [ProcExecWrkr-1] master.SplitLogManager(455): dead splitlog workers [asf911.gq1.ygridcore.net,36099,1515828402164] 2018-01-13 07:26:52,680 INFO [ProcExecWrkr-1] master.SplitLogManager(235): Started splitting 1 logs in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting] for [asf911.gq1.ygridcore.net,36099,1515828402164] 2018-01-13 07:26:52,709 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:52,709 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:52,709 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(331): put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,710 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(363): task not yet acquired /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta ver = 0 2018-01-13 07:26:52,710 DEBUG [SplitLogWorker-asf911:59697] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,59697,1515828407699 is ready to take more tasks, will get task list and try grab tasks again. 2018-01-13 07:26:52,710 DEBUG [SplitLogWorker-asf911:49237] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,49237,1515828402217 is ready to take more tasks, will get task list and try grab tasks again. 2018-01-13 07:26:52,721 INFO [SplitLogWorker-asf911:59697] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,59697,1515828407699 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,721 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,729 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(453): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta acquired by asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:52,744 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter(251): Splitting WAL=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta, length=12886 2018-01-13 07:26:52,754 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,754 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,754 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,755 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta after 1ms 2018-01-13 07:26:52,765 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter(284): DLS Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 2018-01-13 07:26:52,767 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter(336): Finishing writing output logs and closing down 2018-01-13 07:26:52,767 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-01-13 07:26:52,772 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1502): Creating writer path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta.temp 2018-01-13 07:26:52,774 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-01-13 07:26:52,784 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741928_1106{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:52,784 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741928_1106{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:52,785 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1379): Closed wap hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta.temp (wrote 21 edits, skipped 0 edits in 8ms 2018-01-13 07:26:52,787 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1405): Rename hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000005-asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta.temp to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000025 2018-01-13 07:26:52,787 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] wal.WALSplitter(357): Processed 21 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta, length=12886, corrupted=false, progress failed=false 2018-01-13 07:26:52,796 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,796 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,796 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta to final state DONE asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:52,796 INFO [RS_LOG_REPLAY_OPS-asf911:59697-0] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,59697,1515828407699 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@749a9924 in 75ms. Status = DONE 2018-01-13 07:26:52,796 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(371): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta entered state: DONE asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:52,802 INFO [main-EventThread] wal.WALSplitter(449): Archived processed log hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,802 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(414): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,807 WARN [ProcExecWrkr-1] master.SplitLogManager(273): Returning success without actually splitting and deleting all the log files in path hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting: [FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397; isDirectory=false; length=874; replication=2; blocksize=134217728; modification_time=1515828410563; access_time=1515828404399; owner=jenkins.hfs.14; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting is non empty': Directory is not empty at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:84) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3676) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:623) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy29.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy32.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:267) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:293) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:232) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:224) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:117) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:51) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1456) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1225) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1735) 2018-01-13 07:26:52,808 INFO [ProcExecWrkr-1] master.SplitLogManager(287): finished splitting (more than or equal to) 12886 bytes in 1 log files in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting] in 127ms 2018-01-13 07:26:52,812 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,812 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,812 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(694): deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.meta.1515828404621.meta 2018-01-13 07:26:52,812 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:52,813 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:52,870 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=14, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:52,871 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=14, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:52,871 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=15, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164}] 2018-01-13 07:26:52,912 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=15, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:52,912 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=15, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164; forceNewPlan=false, retain=false 2018-01-13 07:26:53,069 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:53,078 INFO [ProcExecWrkr-3] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,090 INFO [ProcExecWrkr-3] assignment.RegionTransitionProcedure(208): Dispatch pid=15, ppid=14, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,242 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,59697,1515828407699 version=3145728 2018-01-13 07:26:53,243 DEBUG [RSProcedureDispatcher-pool83-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,243 DEBUG [RSProcedureDispatcher-pool83-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:26:53,246 INFO [RS-EventLoopGroup-36-11] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:53,246 INFO [RS-EventLoopGroup-36-11] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34200 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:53,247 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59697] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:53,247 INFO [RS_OPEN_META-asf911:59697-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:53,251 INFO [RS_OPEN_META-asf911:59697-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C59697%2C1515828407699.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:53,256 DEBUG [RS-EventLoopGroup-36-12] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:53,256 DEBUG [RS-EventLoopGroup-36-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:53,259 INFO [RS_OPEN_META-asf911:59697-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699/asf911.gq1.ygridcore.net%2C59697%2C1515828407699.meta.1515828413251.meta 2018-01-13 07:26:53,260 DEBUG [RS_OPEN_META-asf911:59697-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:53,260 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:53,260 DEBUG [RS_OPEN_META-asf911:59697-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:53,261 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:53,261 INFO [RS_OPEN_META-asf911:59697-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:53,261 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:53,261 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:53,266 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:53,266 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:53,266 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:53,267 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:53,267 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:53,269 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:53,269 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:53,269 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:53,269 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:53,270 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:53,272 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:53,272 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:53,272 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:53,273 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:53,273 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:53,275 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:53,275 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:53,276 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:53,276 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:53,276 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:53,278 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:53,278 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:53,279 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:53,279 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=2, currentSize=768240, freeSize=1043194064, maxSize=1043962304, heapSize=768240, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:53,279 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:53,283 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(4350): Found 1 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:53,284 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(4447): Replaying edits from hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000025 2018-01-13 07:26:53,296 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(4634): Applied 75, skipped 0, firstSequenceIdInLog=5, maxSequenceIdInLog=25, path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000025 2018-01-13 07:26:53,296 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(2516): Flushing 5/5 column families, memstore=11.18 KB 2018-01-13 07:26:53,311 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741930_1108{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:53,311 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741930_1108{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:53,312 INFO [RS_OPEN_META-asf911:59697-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=25, memsize=10.9 K, hasBloomFilter=false, into tmp file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/info/42584ad437b64752905de251f4b8fa3d 2018-01-13 07:26:53,333 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741931_1109{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:53,333 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741931_1109{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:53,334 INFO [RS_OPEN_META-asf911:59697-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=25, memsize=296, hasBloomFilter=false, into tmp file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/table/ced9dcc664ce4c0a971ea91631bc61df 2018-01-13 07:26:53,344 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/info/42584ad437b64752905de251f4b8fa3d as hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info/42584ad437b64752905de251f4b8fa3d 2018-01-13 07:26:53,352 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HStore(1028): Added hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info/42584ad437b64752905de251f4b8fa3d, entries=70, sequenceid=25, filesize=15.8 K 2018-01-13 07:26:53,354 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/.tmp/table/ced9dcc664ce4c0a971ea91631bc61df as hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table/ced9dcc664ce4c0a971ea91631bc61df 2018-01-13 07:26:53,362 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HStore(1028): Added hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table/ced9dcc664ce4c0a971ea91631bc61df, entries=5, sequenceid=25, filesize=4.9 K 2018-01-13 07:26:53,363 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(2712): Finished memstore flush of ~11.18 KB/11452, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 66ms, sequenceid=25, compaction requested=false; wal=null 2018-01-13 07:26:53,363 INFO [IPC Server handler 3 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741928_1106 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:53,364 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(4427): Deleted recovered.edits file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/0000000000000000025 2018-01-13 07:26:53,366 DEBUG [RS_OPEN_META-asf911:59697-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:53,370 DEBUG [RS_OPEN_META-asf911:59697-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/26.seqid, newSeqId=26, maxSeqId=3 2018-01-13 07:26:53,371 INFO [RS_OPEN_META-asf911:59697-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=26 2018-01-13 07:26:53,374 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:53,377 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=40421] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=26, pid=15, ppid=14, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,377 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:53,377 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828413377}]},"ts":1515828413377} 2018-01-13 07:26:53,379 DEBUG [RS_OPEN_META-asf911:59697-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,385 DEBUG [hconnection-0x1d9fbd5d-shared-pool84-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:26:53,387 INFO [RS-EventLoopGroup-36-17] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:53,387 INFO [RS-EventLoopGroup-36-17] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34208 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:53,391 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:53,392 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,448 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=15, ppid=14, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:53,489 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=14, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:53,490 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1250): Finished pid=15, ppid=14, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,36099,1515828402164 in 536msec 2018-01-13 07:26:53,632 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=14, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.0160sec 2018-01-13 07:26:53,633 INFO [M:1;asf911:40421] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,649 INFO [M:1;asf911:40421] master.MasterWalManager(185): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting doesn't belong to a known region server, splitting 2018-01-13 07:26:53,650 INFO [M:1;asf911:40421] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217 belongs to an existing region server 2018-01-13 07:26:53,650 INFO [M:1;asf911:40421] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699 belongs to an existing region server 2018-01-13 07:26:53,651 INFO [M:1;asf911:40421] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:53,663 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 59e7f350fc3769c7c320345d67c21835, NAME => 'hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.', STARTKEY => '', ENDKEY => ''} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,49237,1515828402217 regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:53,663 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 02aa6cdb76f8c3ab882fdc1614996b03, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.', STARTKEY => '', ENDKEY => '1'} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,49237,1515828402217 regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:53,663 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => b4402e59bcc9a4f34ad3070698f03ed5, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.', STARTKEY => '1', ENDKEY => '2'} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,36099,1515828402164 regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:53,664 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => b15455530ee6b30edbc7069733984b38, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.', STARTKEY => '2', ENDKEY => '3'} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,49237,1515828402217 regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:53,664 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 77233b6b05a0a35d88c2b7315948add1, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.', STARTKEY => '3', ENDKEY => '4'} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,59697,1515828407699 regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,665 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => aed90a6304ebc698cae059a0ac39bff6, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.', STARTKEY => '4', ENDKEY => '5'} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,59697,1515828407699 regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:53,665 INFO [M:1;asf911:40421] assignment.RegionStateStore(135): Load hbase:meta entry region={ENCODED => 2436b98bf1d1a818c8c836b539d7c151, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.', STARTKEY => '5', ENDKEY => ''} regionState=OPEN lastHost=asf911.gq1.ygridcore.net,36099,1515828402164 regionLocation=asf911.gq1.ygridcore.net,36099,1515828402164 2018-01-13 07:26:53,667 INFO [M:1;asf911:40421] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:53,668 INFO [M:1;asf911:40421] assignment.AssignmentManager(1321): KILL RegionServer=asf911.gq1.ygridcore.net,36099,1515828402164 hosting regions but not online. 2018-01-13 07:26:53,668 WARN [M:1;asf911:40421] master.ServerManager(615): Expiration of asf911.gq1.ygridcore.net,36099,1515828402164 but server not online 2018-01-13 07:26:53,668 INFO [M:1;asf911:40421] master.ServerManager(600): Processing expiration of asf911.gq1.ygridcore.net,36099,1515828402164 on asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:53,811 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:53,812 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:26:53,837 DEBUG [M:1;asf911:40421] procedure2.ProcedureExecutor(869): Stored pid=16, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,837 DEBUG [M:1;asf911:40421] assignment.AssignmentManager(1393): Added=asf911.gq1.ygridcore.net,36099,1515828402164 to dead servers, submitted shutdown handler to be executed meta=false 2018-01-13 07:26:53,840 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828418837 2018-01-13 07:26:53,840 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=16, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,840 INFO [ProcExecWrkr-6] procedure.ServerCrashProcedure(118): Start pid=16, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,840 INFO [M:1;asf911:40421] assignment.AssignmentManager(1219): Joined the cluster in 189msec, failover=true 2018-01-13 07:26:53,847 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(355): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:53,854 DEBUG [M:1;asf911:40421] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:49237 2018-01-13 07:26:53,856 INFO [RS-EventLoopGroup-35-27] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:53,856 INFO [RS-EventLoopGroup-35-27] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40815 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:53,873 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:26:53,882 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:53,888 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=16, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,890 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:26:53,898 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:53,899 INFO [M:1;asf911:40421] master.HMaster(928): Master has completed initialization 3.625sec 2018-01-13 07:26:53,900 INFO [M:1;asf911:40421] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:26:53,900 INFO [M:1;asf911:40421] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:26:53,901 INFO [M:1;asf911:40421] regionserver.HRegionServer(866): ClusterId : e753eb97-6687-4d2f-938f-4ade98149f69 2018-01-13 07:26:53,953 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=16, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,953 DEBUG [ProcExecWrkr-7] procedure.ServerCrashProcedure(218): Splitting WALs pid=16, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:53,955 INFO [ProcExecWrkr-7] master.SplitLogManager(455): dead splitlog workers [asf911.gq1.ygridcore.net,36099,1515828402164] 2018-01-13 07:26:53,957 INFO [ProcExecWrkr-7] master.SplitLogManager(235): Started splitting 1 logs in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting] for [asf911.gq1.ygridcore.net,36099,1515828402164] 2018-01-13 07:26:53,965 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:53,965 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(331): put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:53,965 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:53,966 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(363): task not yet acquired /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 ver = 0 2018-01-13 07:26:54,282 DEBUG [SplitLogWorker-asf911:59697] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,59697,1515828407699 is ready to take more tasks, will get task list and try grab tasks again. 2018-01-13 07:26:54,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,290 INFO [SplitLogWorker-asf911:59697] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,59697,1515828407699 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,291 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(453): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 acquired by asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,304 INFO [asf911.gq1.ygridcore.net,40421,1515828402111_splitLogManager__Chore_1] master.SplitLogManager$TimeoutMonitor(585): total=1, unassigned=0, tasks={/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397=last_update = 1515828414291 last_version = 1 cur_worker_name = asf911.gq1.ygridcore.net,59697,1515828407699 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2018-01-13 07:26:54,315 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter(251): Splitting WAL=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397, length=874 2018-01-13 07:26:54,324 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,324 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 2018-01-13 07:26:54,324 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,325 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 after 1ms 2018-01-13 07:26:54,335 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter(284): DLS Last flushed sequenceid for 2436b98bf1d1a818c8c836b539d7c151: last_flushed_sequence_id: 18446744073709551615 2018-01-13 07:26:54,336 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter(284): DLS Last flushed sequenceid for b4402e59bcc9a4f34ad3070698f03ed5: last_flushed_sequence_id: 18446744073709551615 2018-01-13 07:26:54,336 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter(336): Finishing writing output logs and closing down 2018-01-13 07:26:54,336 DEBUG [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-01-13 07:26:54,338 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-01-13 07:26:54,338 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] wal.WALSplitter(357): Processed 0 edits across 0 regions; edits skipped=2; log file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397, length=874, corrupted=false, progress failed=false 2018-01-13 07:26:54,348 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,348 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,349 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 to final state DONE asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,349 INFO [RS_LOG_REPLAY_OPS-asf911:59697-1] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,59697,1515828407699 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@6498bb76 in 58ms. Status = DONE 2018-01-13 07:26:54,349 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(371): task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 entered state: DONE asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,354 INFO [main-EventThread] wal.WALSplitter(449): Archived processed log hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs/asf911.gq1.ygridcore.net%2C36099%2C1515828402164.1515828404397 2018-01-13 07:26:54,355 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(414): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,357 INFO [ProcExecWrkr-7] master.SplitLogManager(287): finished splitting (more than or equal to) 874 bytes in 1 log files in [hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,36099,1515828402164-splitting] in 400ms 2018-01-13 07:26:54,359 DEBUG [ProcExecWrkr-7] procedure.ServerCrashProcedure(226): Done splitting WALs pid=16, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:54,365 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,365 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,365 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:54,365 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(694): deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C36099%2C1515828402164-splitting%2Fasf911.gq1.ygridcore.net%252C36099%252C1515828402164.1515828404397 2018-01-13 07:26:54,392 DEBUG [hconnection-0x51a18b21-metaLookup-shared--pool37-t5] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:26:54,394 INFO [RS-EventLoopGroup-36-19] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:54,394 INFO [RS-EventLoopGroup-36-19] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34213 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:54,398 DEBUG [hconnection-0x51a18b21-shared-pool12-t10] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:54,400 DEBUG [Default-IPC-NioEventLoopGroup-7-32] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:36099 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:36099 2018-01-13 07:26:54,432 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=16, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:54,435 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5}, {pid=18, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151}] 2018-01-13 07:26:54,519 DEBUG [hconnection-0x51a18b21-shared-pool12-t10] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:36099 this server is in the failed servers list 2018-01-13 07:26:54,527 INFO [ProcExecWrkr-10] procedure.MasterProcedureScheduler(883): pid=17, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,528 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=18, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151 testLogSplittingAfterMasterRecoveryDueToZKExpiry testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,530 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=18, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164; forceNewPlan=false, retain=true 2018-01-13 07:26:54,530 INFO [ProcExecWrkr-10] assignment.AssignProcedure(211): Start pid=17, ppid=16, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36099,1515828402164; forceNewPlan=false, retain=true 2018-01-13 07:26:54,682 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 2 regions. 2 retained the pre-restart assignment. 2018-01-13 07:26:54,690 INFO [ProcExecWrkr-13] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,690 INFO [ProcExecWrkr-12] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,692 INFO [ProcExecWrkr-13] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5f505642 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:54,705 DEBUG [ProcExecWrkr-13] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5197b3d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:54,708 DEBUG [MultiHConnection-shared--pool88-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:26:54,712 INFO [RS-EventLoopGroup-36-21] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:54,713 INFO [RS-EventLoopGroup-36-21] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 34219 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:54,723 INFO [ProcExecWrkr-13] assignment.RegionTransitionProcedure(208): Dispatch pid=18, ppid=16, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,724 INFO [ProcExecWrkr-12] assignment.RegionTransitionProcedure(208): Dispatch pid=17, ppid=16, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,728 DEBUG [hconnection-0x51a18b21-shared-pool12-t10] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:36099 this server is in the failed servers list 2018-01-13 07:26:54,876 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,49237,1515828402217 version=3145728 2018-01-13 07:26:54,876 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,59697,1515828407699 version=3145728 2018-01-13 07:26:54,877 DEBUG [RSProcedureDispatcher-pool83-t2] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,877 DEBUG [RSProcedureDispatcher-pool83-t2] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:49237 2018-01-13 07:26:54,878 INFO [RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=59697] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,879 INFO [RS-EventLoopGroup-35-28] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:54,880 INFO [RS-EventLoopGroup-35-28] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 40826 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:54,880 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=49237] regionserver.RSRpcServices(1907): Open testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,883 DEBUG [RS_OPEN_REGION-asf911:59697-2] regionserver.HRegion(6949): Opening region: {ENCODED => b4402e59bcc9a4f34ad3070698f03ed5, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.', STARTKEY => '1', ENDKEY => '2'} 2018-01-13 07:26:54,883 DEBUG [RS_OPEN_REGION-asf911:59697-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry b4402e59bcc9a4f34ad3070698f03ed5 2018-01-13 07:26:54,883 DEBUG [RS_OPEN_REGION-asf911:59697-2] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,884 DEBUG [RS_OPEN_REGION-asf911:49237-2] regionserver.HRegion(6949): Opening region: {ENCODED => 2436b98bf1d1a818c8c836b539d7c151, NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.', STARTKEY => '5', ENDKEY => ''} 2018-01-13 07:26:54,884 DEBUG [RS_OPEN_REGION-asf911:49237-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2436b98bf1d1a818c8c836b539d7c151 2018-01-13 07:26:54,884 DEBUG [RS_OPEN_REGION-asf911:49237-2] regionserver.HRegion(807): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,887 DEBUG [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/col 2018-01-13 07:26:54,887 DEBUG [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/col 2018-01-13 07:26:54,888 DEBUG [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/col 2018-01-13 07:26:54,887 DEBUG [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/col 2018-01-13 07:26:54,888 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:54,888 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:26:54,888 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:54,888 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:26:54,888 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:54,889 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] hfile.CacheConfig(239): Created cacheConfig for col: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:54,889 INFO [StoreOpener-2436b98bf1d1a818c8c836b539d7c151-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:54,889 INFO [StoreOpener-b4402e59bcc9a4f34ad3070698f03ed5-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:54,890 INFO [RS_OPEN_REGION-asf911:49237-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,890 INFO [RS_OPEN_REGION-asf911:59697-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,892 DEBUG [RS_OPEN_REGION-asf911:49237-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151 2018-01-13 07:26:54,892 DEBUG [RS_OPEN_REGION-asf911:59697-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5 2018-01-13 07:26:54,894 DEBUG [RS_OPEN_REGION-asf911:59697-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:54,895 DEBUG [RS_OPEN_REGION-asf911:49237-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testLogSplittingAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:26:54,899 DEBUG [RS_OPEN_REGION-asf911:59697-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/b4402e59bcc9a4f34ad3070698f03ed5/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:54,899 DEBUG [RS_OPEN_REGION-asf911:49237-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testLogSplittingAfterMasterRecoveryDueToZKExpiry/2436b98bf1d1a818c8c836b539d7c151/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:54,900 INFO [RS_OPEN_REGION-asf911:59697-2] regionserver.HRegion(960): Onlined b4402e59bcc9a4f34ad3070698f03ed5; next sequenceid=3 2018-01-13 07:26:54,901 INFO [RS_OPEN_REGION-asf911:49237-2] regionserver.HRegion(960): Onlined 2436b98bf1d1a818c8c836b539d7c151; next sequenceid=3 2018-01-13 07:26:54,902 INFO [PostOpenDeployTasks:b4402e59bcc9a4f34ad3070698f03ed5] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,902 INFO [PostOpenDeployTasks:2436b98bf1d1a818c8c836b539d7c151] regionserver.HRegionServer(2197): Post open deploy tasks for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=17, ppid=16, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5; rit=OPENING, location=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=18, ppid=16, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151; rit=OPENING, location=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,903 DEBUG [PostOpenDeployTasks:b4402e59bcc9a4f34ad3070698f03ed5] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:54,903 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5., regionState=OPEN, openSeqNum=3, regionLocation=asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,911 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151., regionState=OPEN, openSeqNum=3, regionLocation=asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,911 DEBUG [RS_OPEN_REGION-asf911:59697-2] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. on asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:54,911 DEBUG [PostOpenDeployTasks:2436b98bf1d1a818c8c836b539d7c151] regionserver.HRegionServer(2221): Finished post open deploy task for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:54,912 DEBUG [RS_OPEN_REGION-asf911:49237-2] handler.OpenRegionHandler(128): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. on asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:54,982 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=17, ppid=16, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5 2018-01-13 07:26:54,982 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=18, ppid=16, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151 2018-01-13 07:26:54,982 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=17, ppid=16, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=b4402e59bcc9a4f34ad3070698f03ed5 in 497msec 2018-01-13 07:26:55,038 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at '2', inclusive 2018-01-13 07:26:55,043 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at '3', inclusive 2018-01-13 07:26:55,044 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=16, state=RUNNABLE:SERVER_CRASH_FINISH; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false; resume parent processing. 2018-01-13 07:26:55,044 DEBUG [ProcExecWrkr-15] procedure2.StateMachineProcedure(180): pid=16, state=RUNNABLE:SERVER_CRASH_FINISH; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false 2018-01-13 07:26:55,044 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1250): Finished pid=18, ppid=16, state=SUCCESS; AssignProcedure table=testLogSplittingAfterMasterRecoveryDueToZKExpiry, region=2436b98bf1d1a818c8c836b539d7c151 in 497msec 2018-01-13 07:26:55,048 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at '4', inclusive 2018-01-13 07:26:55,053 DEBUG [Time-limited test] client.ClientScanner(242): Advancing internal scanner to startKey at '5', inclusive 2018-01-13 07:26:55,059 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,36099,1515828402164: killAll ***** 2018-01-13 07:26:55,060 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:55,067 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 918557064 }, "NonHeapMemoryUsage" : { "committed" : 138346496, "init" : 2555904, "max" : -1, "used" : 136431168 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 80041, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 323, "TotalCallTime_min" : 0, "TotalCallTime_max" : 14, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 5, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 14, "TotalCallTime_99th_percentile" : 14, "TotalCallTime_99.9th_percentile" : 14, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 14, "TotalCallTime_TimeRangeCount_3-10" : 13, "TotalCallTime_TimeRangeCount_10-30" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 268, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 323, "ResponseSize_min" : 0, "ResponseSize_max" : 1007, "ResponseSize_mean" : 77, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 92, "ResponseSize_95th_percentile" : 764, "ResponseSize_98th_percentile" : 872, "ResponseSize_99th_percentile" : 999, "ResponseSize_99.9th_percentile" : 1006, "ResponseSize_SizeRangeCount_0-10" : 36, "ResponseSize_SizeRangeCount_10-100" : 14, "ResponseSize_SizeRangeCount_100-1000" : 5, "ResponseSize_SizeRangeCount_100000000-inf" : 268, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 323, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 14, "ProcessCallTime_mean" : 3, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 4, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 14, "ProcessCallTime_99th_percentile" : 14, "ProcessCallTime_99.9th_percentile" : 14, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 12, "ProcessCallTime_TimeRangeCount_10-30" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 50, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 323, "RequestSize_min" : 70, "RequestSize_max" : 2917, "RequestSize_mean" : 361, "RequestSize_25th_percentile" : 113, "RequestSize_median" : 154, "RequestSize_75th_percentile" : 692, "RequestSize_90th_percentile" : 1380, "RequestSize_95th_percentile" : 2148, "RequestSize_98th_percentile" : 2609, "RequestSize_99th_percentile" : 2763, "RequestSize_99.9th_percentile" : 2901, "RequestSize_SizeRangeCount_10-100" : 9, "RequestSize_SizeRangeCount_100-1000" : 46, "RequestSize_SizeRangeCount_100000000-inf" : 268, "sentBytes" : 35395, "QueueCallTime_num_ops" : 323, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 55, "QueueCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:55,068 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,49237,1515828402217: killAll ***** 2018-01-13 07:26:55,068 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:55,070 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 918557064 }, "NonHeapMemoryUsage" : { "committed" : 138477568, "init" : 2555904, "max" : -1, "used" : 136532568 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 80041, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 323, "TotalCallTime_min" : 0, "TotalCallTime_max" : 14, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 5, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 14, "TotalCallTime_99th_percentile" : 14, "TotalCallTime_99.9th_percentile" : 14, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 14, "TotalCallTime_TimeRangeCount_3-10" : 13, "TotalCallTime_TimeRangeCount_10-30" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 268, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 323, "ResponseSize_min" : 0, "ResponseSize_max" : 1007, "ResponseSize_mean" : 77, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 92, "ResponseSize_95th_percentile" : 764, "ResponseSize_98th_percentile" : 872, "ResponseSize_99th_percentile" : 999, "ResponseSize_99.9th_percentile" : 1006, "ResponseSize_SizeRangeCount_0-10" : 36, "ResponseSize_SizeRangeCount_10-100" : 14, "ResponseSize_SizeRangeCount_100-1000" : 5, "ResponseSize_SizeRangeCount_100000000-inf" : 268, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 323, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 14, "ProcessCallTime_mean" : 3, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 4, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 14, "ProcessCallTime_99th_percentile" : 14, "ProcessCallTime_99.9th_percentile" : 14, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 12, "ProcessCallTime_TimeRangeCount_10-30" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 50, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 323, "RequestSize_min" : 70, "RequestSize_max" : 2917, "RequestSize_mean" : 361, "RequestSize_25th_percentile" : 113, "RequestSize_median" : 154, "RequestSize_75th_percentile" : 692, "RequestSize_90th_percentile" : 1380, "RequestSize_95th_percentile" : 2148, "RequestSize_98th_percentile" : 2609, "RequestSize_99th_percentile" : 2763, "RequestSize_99.9th_percentile" : 2901, "RequestSize_SizeRangeCount_10-100" : 9, "RequestSize_SizeRangeCount_100-1000" : 46, "RequestSize_SizeRangeCount_100000000-inf" : 268, "sentBytes" : 35395, "QueueCallTime_num_ops" : 323, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 55, "QueueCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:55,073 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,49237,1515828402217 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,49237,1515828402217: killAll ***** 2018-01-13 07:26:55,073 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,49237,1515828402217' ***** 2018-01-13 07:26:55,073 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:55,074 INFO [RS:1;asf911:49237] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:55,074 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,59697,1515828407699: killAll ***** 2018-01-13 07:26:55,074 INFO [RS:1;asf911:49237] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:55,074 INFO [SplitLogWorker-asf911:49237] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:55,074 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:55,074 INFO [RS:1;asf911:49237] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:55,076 INFO [RS:1;asf911:49237] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:55,074 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:55,076 INFO [SplitLogWorker-asf911:49237] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,49237,1515828402217 exiting 2018-01-13 07:26:55,076 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-0] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-1] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-2] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-1] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.: disabling compactions & flushes 2018-01-13 07:26:55,077 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:55,078 DEBUG [RS:1;asf911:49237] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:55,078 DEBUG [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.: disabling compactions & flushes 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-1] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:55,077 DEBUG [RS_CLOSE_REGION-asf911:49237-2] regionserver.HRegion(1481): Closing hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.: disabling compactions & flushes 2018-01-13 07:26:55,078 INFO [RS:1;asf911:49237] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x294963b3 to localhost:59094 2018-01-13 07:26:55,078 DEBUG [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:55,078 DEBUG [RS_CLOSE_REGION-asf911:49237-2] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:55,078 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1308622848, "init" : 792723456, "max" : 2609905664, "used" : 918749600 }, "NonHeapMemoryUsage" : { "committed" : 138739712, "init" : 2555904, "max" : -1, "used" : 136537408 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 80041, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 323, "TotalCallTime_min" : 0, "TotalCallTime_max" : 14, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 5, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 14, "TotalCallTime_99th_percentile" : 14, "TotalCallTime_99.9th_percentile" : 14, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 14, "TotalCallTime_TimeRangeCount_3-10" : 13, "TotalCallTime_TimeRangeCount_10-30" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 268, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 323, "ResponseSize_min" : 0, "ResponseSize_max" : 1007, "ResponseSize_mean" : 77, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 9, "ResponseSize_75th_percentile" : 56, "ResponseSize_90th_percentile" : 92, "ResponseSize_95th_percentile" : 764, "ResponseSize_98th_percentile" : 872, "ResponseSize_99th_percentile" : 999, "ResponseSize_99.9th_percentile" : 1006, "ResponseSize_SizeRangeCount_0-10" : 36, "ResponseSize_SizeRangeCount_10-100" : 14, "ResponseSize_SizeRangeCount_100-1000" : 5, "ResponseSize_SizeRangeCount_100000000-inf" : 268, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 323, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 14, "ProcessCallTime_mean" : 3, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 4, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 14, "ProcessCallTime_99th_percentile" : 14, "ProcessCallTime_99.9th_percentile" : 14, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 15, "ProcessCallTime_TimeRangeCount_3-10" : 12, "ProcessCallTime_TimeRangeCount_10-30" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 50, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 323, "RequestSize_min" : 70, "RequestSize_max" : 2917, "RequestSize_mean" : 361, "RequestSize_25th_percentile" : 113, "RequestSize_median" : 154, "RequestSize_75th_percentile" : 692, "RequestSize_90th_percentile" : 1380, "RequestSize_95th_percentile" : 2148, "RequestSize_98th_percentile" : 2609, "RequestSize_99th_percentile" : 2763, "RequestSize_99.9th_percentile" : 2901, "RequestSize_SizeRangeCount_10-100" : 9, "RequestSize_SizeRangeCount_100-1000" : 46, "RequestSize_SizeRangeCount_100000000-inf" : 268, "sentBytes" : 35395, "QueueCallTime_num_ops" : 323, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 55, "QueueCallTime_TimeRangeCount_600000-inf" : 268, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043194064, "blockCacheCount" : 2, "blockCacheSize" : 768240, "blockCacheHitCount" : 1, "blockCacheHitCountPrimary" : 1, "blockCacheMissCount" : 2, "blockCacheMissCountPrimary" : 2, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 33.33333333333333, "blockCacheExpressHitPercent" : 33.33333333333333, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 2, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 1, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 1, "l1CacheMissCount" : 2, "l1CacheHitRatio" : 0.3333333333333333, "l1CacheMissRatio" : 0.6666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:26:55,079 INFO [StoreCloserThread-hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835.-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:55,079 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,078 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,083 INFO [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:55,078 DEBUG [RS:1;asf911:49237] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,083 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40421] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,59697,1515828407699 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,59697,1515828407699: killAll ***** 2018-01-13 07:26:55,083 DEBUG [RS_CLOSE_REGION-asf911:49237-0] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1515828407891.b15455530ee6b30edbc7069733984b38. 2018-01-13 07:26:55,083 ERROR [RS_CLOSE_REGION-asf911:49237-1] regionserver.HRegion(1620): Memstore size is 210 2018-01-13 07:26:55,081 ERROR [RS_CLOSE_REGION-asf911:49237-2] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:26:55,086 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,59697,1515828407699' ***** 2018-01-13 07:26:55,087 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:26:55,085 INFO [RS_CLOSE_REGION-asf911:49237-1] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:55,084 DEBUG [RS_CLOSE_REGION-asf911:49237-0] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:55,084 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1388): Waiting on 3 regions to close 2018-01-13 07:26:55,088 DEBUG [RS_CLOSE_REGION-asf911:49237-1] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1515828407891.02aa6cdb76f8c3ab882fdc1614996b03. 2018-01-13 07:26:55,088 INFO [RS:2;asf911:59697] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:26:55,088 ERROR [main] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:26:55,090 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:55,090 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,40421,1515828402111' ***** 2018-01-13 07:26:55,090 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:26:55,087 INFO [RS_CLOSE_REGION-asf911:49237-2] regionserver.HRegion(1633): Closed hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:55,090 INFO [SplitLogWorker-asf911:59697] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:26:55,090 INFO [RS:2;asf911:59697] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:26:55,089 DEBUG [RS:1;asf911:49237] regionserver.HRegionServer(1392): Online Regions={59e7f350fc3769c7c320345d67c21835=hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835., 2436b98bf1d1a818c8c836b539d7c151=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.} 2018-01-13 07:26:55,091 INFO [RS:2;asf911:59697] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:26:55,092 INFO [RS:2;asf911:59697] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:26:55,089 DEBUG [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.: disabling compactions & flushes 2018-01-13 07:26:55,093 DEBUG [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:55,092 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:26:55,091 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:26:55,091 DEBUG [M:1;asf911:40421] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bba9b15, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:55,091 INFO [SplitLogWorker-asf911:59697] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,59697,1515828407699 exiting 2018-01-13 07:26:55,091 DEBUG [RS_CLOSE_REGION-asf911:49237-2] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828404867.59e7f350fc3769c7c320345d67c21835. 2018-01-13 07:26:55,093 INFO [M:1;asf911:40421] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:55,094 DEBUG [M:1;asf911:40421] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:55,094 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,094 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:55,094 INFO [RS_CLOSE_REGION-asf911:49237-0] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:55,094 DEBUG [RS_CLOSE_REGION-asf911:59697-1] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:55,094 DEBUG [RS_CLOSE_REGION-asf911:59697-0] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:55,096 DEBUG [RS_CLOSE_REGION-asf911:59697-1] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.: disabling compactions & flushes 2018-01-13 07:26:55,094 DEBUG [RS_CLOSE_REGION-asf911:59697-2] handler.CloseRegionHandler(92): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:55,094 INFO [M:1;asf911:40421] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x367ec517 to localhost:59094 2018-01-13 07:26:55,094 DEBUG [RS_CLOSE_REGION-asf911:49237-0] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1515828407891.2436b98bf1d1a818c8c836b539d7c151. 2018-01-13 07:26:55,094 DEBUG [RS:2;asf911:59697] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:26:55,096 DEBUG [M:1;asf911:40421] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,096 DEBUG [RS_CLOSE_REGION-asf911:59697-2] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.: disabling compactions & flushes 2018-01-13 07:26:55,096 DEBUG [RS_CLOSE_REGION-asf911:59697-2] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:55,096 DEBUG [RS_CLOSE_REGION-asf911:59697-1] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:55,096 DEBUG [RS_CLOSE_REGION-asf911:59697-0] regionserver.HRegion(1481): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.: disabling compactions & flushes 2018-01-13 07:26:55,096 INFO [M:1;asf911:40421] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,40421,1515828402111; all regions closed. 2018-01-13 07:26:55,097 DEBUG [M:1;asf911:40421] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,097 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,096 INFO [RS:2;asf911:59697] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x6f556bba to localhost:59094 2018-01-13 07:26:55,097 INFO [RS_CLOSE_REGION-asf911:59697-1] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:55,097 INFO [M:1;asf911:40421] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,40421,1515828402111 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,40421,1515828402111-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,40421,1515828402111-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,40421,1515828402111-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:40421 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,40421,1515828402111-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,40421,1515828402111-BalancerChore Period: 300000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:55,097 DEBUG [RS_CLOSE_REGION-asf911:59697-0] regionserver.HRegion(1520): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:55,097 INFO [M:1;asf911:40421] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:26:55,098 INFO [M:1;asf911:40421] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:26:55,098 DEBUG [M:1;asf911:40421] master.HMaster(1187): Stopping service threads 2018-01-13 07:26:55,097 DEBUG [RS_CLOSE_REGION-asf911:59697-1] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1515828407891.b4402e59bcc9a4f34ad3070698f03ed5. 2018-01-13 07:26:55,097 DEBUG [RS:2;asf911:59697] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,098 INFO [RS:2;asf911:59697] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:55,098 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,098 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1250): Finished pid=16, state=SUCCESS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,36099,1515828402164, splitWal=true, meta=false in 1.3760sec 2018-01-13 07:26:55,098 INFO [RS:2;asf911:59697] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:55,101 INFO [RS:2;asf911:59697] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:55,098 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1.-1] regionserver.HStore(930): Closed col 2018-01-13 07:26:55,103 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:55,103 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:55,103 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:26:55,101 INFO [RS_CLOSE_REGION-asf911:59697-0] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:55,106 WARN [main] zookeeper.ZKUtil(618): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:55,113 ERROR [main] zookeeper.ZKWatcher(578): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:55,113 ERROR [main] master.ActiveMasterManager(280): master:44597-0x160ee6894500066, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:26:55,114 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,40421,1515828402111 2018-01-13 07:26:55,105 INFO [RS_CLOSE_REGION-asf911:59697-2] regionserver.HRegion(1633): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:55,103 DEBUG [RS_CLOSE_META-asf911:59697-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:26:55,114 DEBUG [RS_CLOSE_REGION-asf911:59697-2] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1515828407891.77233b6b05a0a35d88c2b7315948add1. 2018-01-13 07:26:55,103 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1388): Waiting on 3 regions to close 2018-01-13 07:26:55,116 DEBUG [RS:2;asf911:59697] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-01-13 07:26:55,103 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:55,116 DEBUG [RS_CLOSE_META-asf911:59697-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:55,113 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:55,113 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:55,113 DEBUG [M:1;asf911:40421] zookeeper.ZKUtil(614): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:26:55,117 WARN [M:1;asf911:40421] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:26:55,117 INFO [M:1;asf911:40421] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:26:55,106 DEBUG [RS_CLOSE_REGION-asf911:59697-0] handler.CloseRegionHandler(125): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1515828407891.aed90a6304ebc698cae059a0ac39bff6. 2018-01-13 07:26:55,117 DEBUG [RS_CLOSE_META-asf911:59697-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:55,118 INFO [M:1;asf911:40421] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5f505642 to localhost:59094 2018-01-13 07:26:55,118 DEBUG [M:1;asf911:40421] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,118 INFO [M:1;asf911:40421] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:26:55,120 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:55,120 INFO [M:1;asf911:40421] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:26:55,122 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:55,122 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,122 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,122 INFO [M:1;asf911:40421] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:26:55,124 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,124 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:55,124 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:55,123 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,123 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,123 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,123 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:26:55,125 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:55,124 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:55,128 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:55,128 ERROR [RS_CLOSE_META-asf911:59697-0] regionserver.HRegion(1620): Memstore size is 2668 2018-01-13 07:26:55,128 DEBUG [RS_CLOSE_META-asf911:59697-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:26:55,129 INFO [RS_CLOSE_META-asf911:59697-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:55,129 DEBUG [RS_CLOSE_META-asf911:59697-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:55,129 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741927_1105{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:55,129 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741927_1105{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 214 2018-01-13 07:26:55,130 INFO [M:1;asf911:40421] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,40421,1515828402111_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:55,130 INFO [M:1;asf911:40421] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:26:55,130 INFO [M:1;asf911:40421] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:55,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:55,133 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:40421-0x160ee6894500067, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:55,133 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:55,140 DEBUG [M:1;asf911:40421] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,40421,1515828402111 already deleted, retry=false 2018-01-13 07:26:55,157 INFO [M:1;asf911:40421] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,40421,1515828402111; zookeeper connection closed. 2018-01-13 07:26:55,157 INFO [M:1;asf911:40421] regionserver.HRegionServer(1157): M:1;asf911:40421 exiting 2018-01-13 07:26:55,291 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,49237,1515828402217; all regions closed. 2018-01-13 07:26:55,295 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741915_1093{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 2575 2018-01-13 07:26:55,295 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,49237,1515828402217/asf911.gq1.ygridcore.net%2C49237%2C1515828402217.1515828404397 not finished, retry = 0 2018-01-13 07:26:55,295 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741915_1093 size 2575 2018-01-13 07:26:55,317 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,59697,1515828407699; all regions closed. 2018-01-13 07:26:55,321 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741929_1107{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 3362 2018-01-13 07:26:55,321 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699/asf911.gq1.ygridcore.net%2C59697%2C1515828407699.meta.1515828413251.meta not finished, retry = 0 2018-01-13 07:26:55,322 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741929_1107 size 3362 2018-01-13 07:26:55,338 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:55,338 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:55,341 INFO [asf911.gq1.ygridcore.net,49237,1515828402217_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 07:26:55,397 DEBUG [RS:1;asf911:49237] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,397 INFO [RS:1;asf911:49237] regionserver.Leases(147): RS:1;asf911:49237 closing leases 2018-01-13 07:26:55,397 INFO [RS:1;asf911:49237] regionserver.Leases(150): RS:1;asf911:49237 closed leases 2018-01-13 07:26:55,398 INFO [RS:1;asf911:49237] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,49237,1515828402217 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,49237,1515828402217 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:55,398 INFO [RS:1;asf911:49237] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:26:55,398 INFO [RS:1;asf911:49237] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:26:55,398 INFO [RS:1;asf911:49237] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:26:55,398 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:55,399 INFO [RS:1;asf911:49237] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:55,415 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:55,415 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:49237-0x160ee6894500069, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:55,415 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,49237,1515828402217 2018-01-13 07:26:55,416 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:55,423 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,49237,1515828402217; zookeeper connection closed. 2018-01-13 07:26:55,424 INFO [RS:1;asf911:49237] regionserver.HRegionServer(1157): RS:1;asf911:49237 exiting 2018-01-13 07:26:55,424 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@c1ac0e1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@c1ac0e1 2018-01-13 07:26:55,437 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741926_1104{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 1263 2018-01-13 07:26:55,438 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(841): complete file /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,59697,1515828407699/asf911.gq1.ygridcore.net%2C59697%2C1515828407699.1515828408888 not finished, retry = 0 2018-01-13 07:26:55,438 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741926_1104 size 1263 2018-01-13 07:26:55,539 DEBUG [RS:2;asf911:59697] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:26:55,540 INFO [RS:2;asf911:59697] regionserver.Leases(147): RS:2;asf911:59697 closing leases 2018-01-13 07:26:55,540 INFO [RS:2;asf911:59697] regionserver.Leases(150): RS:2;asf911:59697 closed leases 2018-01-13 07:26:55,540 INFO [RS:2;asf911:59697] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,59697,1515828407699 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,59697,1515828407699 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:26:55,540 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:26:55,541 INFO [RS:2;asf911:59697] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:26:55,548 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:59697-0x160ee6894500071, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59697,1515828407699 2018-01-13 07:26:55,571 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,59697,1515828407699; zookeeper connection closed. 2018-01-13 07:26:55,571 INFO [RS:2;asf911:59697] regionserver.HRegionServer(1157): RS:2;asf911:59697 exiting 2018-01-13 07:26:55,571 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@263dac25] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@263dac25 2018-01-13 07:26:55,572 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 3 regionserver(s) complete 2018-01-13 07:26:55,583 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,585 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741913_1091 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,585 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741927_1105 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,585 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741915_1093 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,585 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741926_1104 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,585 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741929_1107 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741919_1097 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741922_1100 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741921_1099 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741924_1102 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741925_1103 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,586 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741923_1101 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741920_1098 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741912_1090 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741911_1089 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741930_1108 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741931_1109 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741917_1095 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,587 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741918_1096 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,588 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741910_1088 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,588 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741909_1087 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:26:55,588 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741914_1092 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,588 INFO [IPC Server handler 1 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741916_1094 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:26:55,589 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:55,599 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:55,601 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500077 connected 2018-01-13 07:26:55,602 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:26:55,602 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:26:55,603 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:26:55,604 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:26:55,604 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:26:55,605 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:26:55,606 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:26:55,606 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:26:55,607 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:26:55,607 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:26:55,608 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:26:55,608 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:26:55,609 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:26:55,612 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:26:55,612 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:26:55,613 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:26:55,614 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:26:55,615 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:26:55,615 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:26:55,617 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:26:55,617 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:26:55,654 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:26:55,654 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:55,655 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:26:55,656 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:26:55,656 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:26:55,656 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:55,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500077, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:26:55,707 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=643 (was 521) Potentially hanging thread: IPC Parameter Sending Thread #3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-34-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-25 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-26 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-24 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:40421-HFileCleaner.large.0-1515828410413 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-35-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-32-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:40421-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-35-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-34-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.16 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-35-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-25 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:1;asf911:40421-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-36-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.14 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: M:0;asf911:44597-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-36-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-27 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-33-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.14@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-665578734_2960 at /127.0.0.1:55249 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-608613814_1 at /127.0.0.1:41085 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-30 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-32 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-665578734_2960 at /127.0.0.1:55250 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:44597-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-36-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-32-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-28 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-28 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:44597-HFileCleaner.small.0-1515828403074 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: M:1;asf911:40421-HFileCleaner.small.0-1515828410414 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: RS-EventLoopGroup-36-22 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-27 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-32-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.16@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-33-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-29 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-34-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-34-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-33-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-shared-pool12-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-26 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.15 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-36-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-32-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-34-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:44597-HFileCleaner.large.0-1515828403074 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-36-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Time-limited test-SendThread(localhost:59094) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230) org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) Potentially hanging thread: RS-EventLoopGroup-35-23 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.15@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-32-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-7-31 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x51a18b21-metaLookup-shared--pool37-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-35-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-36-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=4187 (was 3629) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 338), ProcessCount=271 (was 270) - ProcessCount LEAK? -, AvailableMemoryMB=10133 (was 10227) 2018-01-13 07:26:55,708 WARN [main] hbase.ResourceChecker(135): Thread=643 is superior to 500 2018-01-13 07:26:55,708 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=4187 is superior to 1024 2018-01-13 07:26:55,752 INFO [main] hbase.ResourceChecker(148): before: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=641, OpenFileDescriptor=4187, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=271, AvailableMemoryMB=10132 2018-01-13 07:26:55,752 WARN [main] hbase.ResourceChecker(135): Thread=641 is superior to 500 2018-01-13 07:26:55,753 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=4187 is superior to 1024 2018-01-13 07:26:55,766 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741932_1110{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:26:55,766 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741932_1110{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:55,768 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 with version=8 2018-01-13 07:26:55,769 INFO [main] hbase.HBaseTestingUtility(1206): The hbase.fs.tmp.dir is set to hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase-staging 2018-01-13 07:26:55,771 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:55,772 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:55,772 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:55,772 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:55,772 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:55,773 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:50890 2018-01-13 07:26:55,773 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,774 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,774 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,776 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,777 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:50890 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:55,804 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:508900x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:55,807 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:50890-0x160ee6894500078 connected 2018-01-13 07:26:55,821 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:26:55,822 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:26:55,835 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:49352 to delete [blk_1073741920_1098, blk_1073741921_1099, blk_1073741922_1100, blk_1073741923_1101, blk_1073741924_1102, blk_1073741925_1103, blk_1073741926_1104, blk_1073741927_1105, blk_1073741928_1106, blk_1073741929_1107, blk_1073741930_1108, blk_1073741931_1109, blk_1073741909_1087, blk_1073741910_1088, blk_1073741911_1089, blk_1073741912_1090, blk_1073741913_1091, blk_1073741914_1092, blk_1073741915_1093, blk_1073741916_1094, blk_1073741917_1095, blk_1073741918_1096, blk_1073741919_1097] 2018-01-13 07:26:55,907 DEBUG [main] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:55,908 DEBUG [main] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:55,909 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=50890 2018-01-13 07:26:55,910 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=50890 2018-01-13 07:26:55,911 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=50890 2018-01-13 07:26:55,912 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:55,914 INFO [main] client.ConnectionUtils(122): master/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:55,914 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:55,914 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:55,914 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:55,915 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:55,916 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:36180 2018-01-13 07:26:55,916 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,916 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,917 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,919 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,920 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:36180 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:55,940 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:361800x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:55,943 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:36180-0x160ee6894500079 connected 2018-01-13 07:26:55,948 DEBUG [main] zookeeper.RecoverableZooKeeper(571): Node /hbase already exists 2018-01-13 07:26:55,952 DEBUG [main] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:55,953 DEBUG [main] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:55,953 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=36180 2018-01-13 07:26:55,954 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=36180 2018-01-13 07:26:55,955 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=36180 2018-01-13 07:26:55,956 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9, hbase.cluster.distributed=false 2018-01-13 07:26:55,980 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:55,981 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:55,981 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:55,981 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:55,981 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:55,981 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:55,982 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:60497 2018-01-13 07:26:55,983 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,983 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:55,984 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,985 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:55,986 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:60497 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:55,999 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:604970x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:56,001 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:60497-0x160ee689450007a connected 2018-01-13 07:26:56,001 DEBUG [main] zookeeper.ZKUtil(357): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:56,002 DEBUG [main] zookeeper.ZKUtil(357): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:56,002 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=60497 2018-01-13 07:26:56,003 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=60497 2018-01-13 07:26:56,004 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=60497 2018-01-13 07:26:56,029 INFO [main] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:26:56,029 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:26:56,029 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:26:56,030 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:26:56,030 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:26:56,030 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:26:56,031 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:34281 2018-01-13 07:26:56,031 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,031 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,032 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:56,034 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:56,035 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:34281 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:56,049 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:342810x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:56,051 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:34281-0x160ee689450007b connected 2018-01-13 07:26:56,051 DEBUG [main] zookeeper.ZKUtil(357): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:26:56,052 DEBUG [main] zookeeper.ZKUtil(357): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 07:26:56,052 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=34281 2018-01-13 07:26:56,054 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=34281 2018-01-13 07:26:56,055 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=34281 2018-01-13 07:26:56,056 INFO [M:0;asf911:50890] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,056 INFO [M:1;asf911:36180] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:26:56,066 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,066 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:26:56,073 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:56,074 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:56,074 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:56,077 DEBUG [M:1;asf911:36180] zookeeper.RecoverableZooKeeper(571): Node /hbase/master already exists 2018-01-13 07:26:56,074 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:26:56,079 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:56,084 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:56,085 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:56,085 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:26:56,086 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:56,086 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:26:56,086 INFO [M:1;asf911:36180] master.ActiveMasterManager(218): Another master is the active master, asf911.gq1.ygridcore.net,50890,1515828415770; waiting to become the next active master 2018-01-13 07:26:56,086 INFO [M:0;asf911:50890] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,50890,1515828415770 from backup master directory 2018-01-13 07:26:56,096 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,096 WARN [M:0;asf911:50890] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:56,096 INFO [M:0;asf911:50890] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,114 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741933_1111{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:56,114 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741933_1111{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:56,116 DEBUG [M:0;asf911:50890] util.FSUtils(665): Created cluster ID file at hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/hbase.id with ID: 57f05237-7556-4025-95c5-f7f004fd1e7d 2018-01-13 07:26:56,121 INFO [M:0;asf911:50890] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 07:26:56,122 INFO [M:0;asf911:50890] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 Table name == hbase:meta 2018-01-13 07:26:56,131 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741934_1112{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:56,132 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741934_1112{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:56,132 DEBUG [M:0;asf911:50890] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:56,136 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:56,137 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:56,137 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,137 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:56,141 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:56,141 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:56,141 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,141 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:56,144 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:56,144 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:56,145 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,145 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:56,148 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:56,148 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:56,148 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,148 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:56,157 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:56,157 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:56,157 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:56,158 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:56,159 DEBUG [M:0;asf911:50890] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:56,162 DEBUG [M:0;asf911:50890] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:56,166 DEBUG [M:0;asf911:50890] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:56,166 INFO [M:0;asf911:50890] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 07:26:56,166 DEBUG [M:0;asf911:50890] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:26:56,166 DEBUG [M:0;asf911:50890] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:26:56,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:26:56,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:26:56,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:26:56,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:26:56,167 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:26:56,167 INFO [M:0;asf911:50890] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:26:56,177 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741935_1113{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:56,178 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741935_1113{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:56,180 DEBUG [M:0;asf911:50890] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:56,208 INFO [M:0;asf911:50890] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:26:56,209 INFO [M:0;asf911:50890] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:26:56,209 DEBUG [M:0;asf911:50890] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:26:56,225 INFO [M:0;asf911:50890] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5e1995ef to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:56,238 DEBUG [M:0;asf911:50890] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d83a9e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:56,257 INFO [M:0;asf911:50890] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:26:56,258 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:26:56,259 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:26:56,266 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:26:56,266 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:26:56,267 INFO [M:0;asf911:50890] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:26:56,279 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:56,279 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:56,279 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:56,279 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 07:26:56,280 INFO [M:0;asf911:50890] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,50890,1515828415770, sessionid=0x160ee6894500078, setting cluster-up flag (Was=false) 2018-01-13 07:26:56,340 INFO [M:0;asf911:50890] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:26:56,342 DEBUG [M:0;asf911:50890] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,387 INFO [M:0;asf911:50890] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:26:56,389 DEBUG [M:0;asf911:50890] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:26:56,390 INFO [M:0;asf911:50890] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:50890, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:50890, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:50890, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:50890, corePoolSize=5, maxPoolSize=5 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:50890, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:56,391 DEBUG [M:0;asf911:50890] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:50890, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:56,393 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:26:56,393 INFO [M:0;asf911:50890] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:26:56,396 DEBUG [M:0;asf911:50890] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 07:26:56,397 INFO [M:0;asf911:50890] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 07:26:56,397 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 4msec 2018-01-13 07:26:56,397 DEBUG [M:0;asf911:50890] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 07:26:56,397 DEBUG [M:0;asf911:50890] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 07:26:56,397 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 0msec 2018-01-13 07:26:56,400 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828446400 2018-01-13 07:26:56,400 INFO [M:0;asf911:50890] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:26:56,401 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:26:56,401 INFO [M:0;asf911:50890] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:26:56,402 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:26:56,402 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:26:56,402 INFO [M:0;asf911:50890] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:26:56,405 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:26:56,405 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:26:56,405 DEBUG [M:0;asf911:50890] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:26:56,405 DEBUG [M:0;asf911:50890] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;asf911:50890-HFileCleaner.large.0-1515828416405,5,main] 2018-01-13 07:26:56,406 DEBUG [M:0;asf911:50890] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;asf911:50890-HFileCleaner.small.0-1515828416406,5,main] 2018-01-13 07:26:56,406 INFO [M:0;asf911:50890] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828416066, completionTime=-1 2018-01-13 07:26:56,406 INFO [M:0;asf911:50890] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=2 server(s), max=2 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 07:26:56,415 DEBUG [M:0;asf911:50890-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:26:56,417 DEBUG [M:0;asf911:50890-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee689450007d connected 2018-01-13 07:26:56,467 INFO [RS:0;asf911:60497] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x5840413c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:56,467 INFO [RS:1;asf911:34281] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x6d6c3f5f to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:56,516 DEBUG [RS:0;asf911:60497] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5687b444, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:56,517 INFO [RS:0;asf911:60497] regionserver.HRegionServer(866): ClusterId : 57f05237-7556-4025-95c5-f7f004fd1e7d 2018-01-13 07:26:56,517 DEBUG [RS:1;asf911:34281] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63abcc82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:56,517 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:56,577 DEBUG [RS:0;asf911:60497] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:56,577 INFO [RS:1;asf911:34281] regionserver.HRegionServer(866): ClusterId : 57f05237-7556-4025-95c5-f7f004fd1e7d 2018-01-13 07:26:56,577 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:26:56,599 DEBUG [RS:1;asf911:34281] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:26:56,599 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:56,599 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:56,624 DEBUG [RS:0;asf911:60497] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:56,624 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:26:56,624 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:26:56,665 DEBUG [RS:1;asf911:34281] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:26:56,665 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:56,666 DEBUG [RS:0;asf911:60497] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52b82a4c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:56,666 DEBUG [RS:0;asf911:60497] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:60497 2018-01-13 07:26:56,666 INFO [RS:0;asf911:60497] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:56,666 INFO [RS:0;asf911:60497] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:56,666 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:26:56,667 DEBUG [RS:1;asf911:34281] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6794542f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:26:56,667 DEBUG [RS:1;asf911:34281] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:34281 2018-01-13 07:26:56,667 INFO [RS:0;asf911:60497] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,50890,1515828415770 with port=60497, startcode=1515828415979 2018-01-13 07:26:56,667 INFO [RS:1;asf911:34281] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:26:56,667 INFO [RS:1;asf911:34281] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:26:56,668 DEBUG [RS:0;asf911:60497] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:26:56,668 INFO [RS:1;asf911:34281] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,50890,1515828415770 with port=34281, startcode=1515828416028 2018-01-13 07:26:56,668 DEBUG [RS:1;asf911:34281] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:26:56,670 INFO [RS-EventLoopGroup-37-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.17 (auth:SIMPLE) 2018-01-13 07:26:56,670 INFO [RS-EventLoopGroup-37-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 41504 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:56,670 INFO [RS-EventLoopGroup-37-3] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.18 (auth:SIMPLE) 2018-01-13 07:26:56,673 INFO [RS-EventLoopGroup-37-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 49061 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:56,674 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=50890] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,677 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,677 DEBUG [RS:0;asf911:60497] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:56,677 DEBUG [RS:1;asf911:34281] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:26:56,678 DEBUG [RS:0;asf911:60497] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:56,678 DEBUG [RS:1;asf911:34281] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:26:56,678 DEBUG [RS:0;asf911:60497] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:56,678 DEBUG [RS:1;asf911:34281] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:26:56,687 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:26:56,688 DEBUG [RS:0;asf911:60497] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,688 WARN [RS:0;asf911:60497] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:56,688 DEBUG [RS:1;asf911:34281] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,688 DEBUG [RS:0;asf911:60497] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,688 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,688 WARN [RS:1;asf911:34281] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:26:56,689 DEBUG [RS:1;asf911:34281] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,689 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,699 INFO [RS:0;asf911:60497] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:56,699 INFO [RS:1;asf911:34281] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:56,700 DEBUG [RS:0;asf911:60497] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,700 DEBUG [RS:1;asf911:34281] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,701 DEBUG [RS:0;asf911:60497] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,701 DEBUG [RS:1;asf911:34281] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,702 INFO [RS:0;asf911:60497] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x359e48ba to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:56,702 INFO [RS:1;asf911:34281] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x560e9e80 to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:56,707 INFO [M:0;asf911:50890] master.ServerManager(891): Finished wait on RegionServer count=2; waited=301ms, expected min=2 server(s), max=2 server(s), master is running 2018-01-13 07:26:56,730 DEBUG [RS:0;asf911:60497] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77954130, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:56,730 INFO [M:0;asf911:50890] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 07:26:56,730 DEBUG [RS:1;asf911:34281] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42b939f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:56,731 DEBUG [RS:1;asf911:34281] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:56,730 DEBUG [RS:0;asf911:60497] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:26:56,731 INFO [RS:1;asf911:34281] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:56,731 INFO [RS:0;asf911:60497] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:26:56,876 INFO [RS:0;asf911:60497] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:56,877 INFO [RS:1;asf911:34281] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:26:56,877 INFO [RS:0;asf911:60497] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:56,877 INFO [RS:1;asf911:34281] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:26:56,878 INFO [RS:0;asf911:60497] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:56,880 INFO [RS:1;asf911:34281] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:26:56,887 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:60497, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,895 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:60497, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:56,895 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:34281, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,895 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:60497, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,895 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:34281, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:56,895 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:60497, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,895 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:34281, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,896 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:60497, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:56,896 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:34281, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,896 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:60497, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:56,896 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:34281, corePoolSize=1, maxPoolSize=1 2018-01-13 07:26:56,896 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:60497, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:56,896 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:34281, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:56,897 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:60497, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,897 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:34281, corePoolSize=10, maxPoolSize=10 2018-01-13 07:26:56,897 DEBUG [RS:0;asf911:60497] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:60497, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:56,897 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:34281, corePoolSize=3, maxPoolSize=3 2018-01-13 07:26:56,897 DEBUG [RS:1;asf911:34281] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:34281, corePoolSize=2, maxPoolSize=2 2018-01-13 07:26:56,933 INFO [RS:0;asf911:60497] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:56,933 INFO [SplitLogWorker-asf911:60497] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,60497,1515828415979 starting 2018-01-13 07:26:56,942 INFO [RS:1;asf911:34281] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:26:56,942 INFO [SplitLogWorker-asf911:34281] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,34281,1515828416028 starting 2018-01-13 07:26:56,959 DEBUG [M:0;asf911:50890] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:56,962 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:56,962 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:56,963 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:56,964 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 07:26:56,965 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,60497,1515828415979, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:60497, sessionid=0x160ee689450007a 2018-01-13 07:26:56,966 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:56,966 DEBUG [RS:0;asf911:60497] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,966 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,60497,1515828415979' 2018-01-13 07:26:56,966 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:56,966 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,34281,1515828416028, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:34281, sessionid=0x160ee689450007b 2018-01-13 07:26:56,966 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:26:56,966 DEBUG [RS:1;asf911:34281] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,966 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,34281,1515828416028' 2018-01-13 07:26:56,966 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:26:56,966 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:56,967 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:26:56,967 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:56,967 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:56,967 DEBUG [RS:0;asf911:60497] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:56,967 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,60497,1515828415979' 2018-01-13 07:26:56,967 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:56,967 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:26:56,968 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:26:56,968 DEBUG [RS:1;asf911:34281] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:26:56,968 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,34281,1515828416028' 2018-01-13 07:26:56,968 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:26:56,968 DEBUG [RS:0;asf911:60497] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:56,968 DEBUG [RS:1;asf911:34281] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:26:56,969 DEBUG [RS:0;asf911:60497] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:56,969 INFO [RS:0;asf911:60497] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:56,969 INFO [RS:0;asf911:60497] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:56,969 DEBUG [RS:1;asf911:34281] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:26:56,969 INFO [RS:1;asf911:34281] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:26:56,969 INFO [RS:1;asf911:34281] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:26:57,015 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:26:57,016 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 07:26:57,016 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 07:26:57,016 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 07:26:57,076 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 07:26:57,076 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 07:26:57,227 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 07:26:57,227 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 07:26:57,234 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:57,973 INFO [RS:0;asf911:60497] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C60497%2C1515828415979, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:57,974 INFO [RS:1;asf911:34281] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C34281%2C1515828416028, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34281,1515828416028, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:58,007 DEBUG [RS-EventLoopGroup-40-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:58,007 DEBUG [RS-EventLoopGroup-40-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:58,008 DEBUG [RS-EventLoopGroup-40-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK] 2018-01-13 07:26:58,007 DEBUG [RS-EventLoopGroup-40-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:26:58,011 INFO [RS:0;asf911:60497] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979/asf911.gq1.ygridcore.net%2C60497%2C1515828415979.1515828417974 2018-01-13 07:26:58,011 DEBUG [RS:0;asf911:60497] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK]] 2018-01-13 07:26:58,011 INFO [RS:1;asf911:34281] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34281,1515828416028/asf911.gq1.ygridcore.net%2C34281%2C1515828416028.1515828417975 2018-01-13 07:26:58,011 DEBUG [RS:1;asf911:34281] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-5ba0c480-698e-4fa2-be8d-55b114c682ac,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:58,014 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,032 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 07:26:58,041 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,193 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,60497,1515828415979 version=3145728 2018-01-13 07:26:58,194 DEBUG [RSProcedureDispatcher-pool89-t1] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,195 DEBUG [RSProcedureDispatcher-pool89-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:60497 2018-01-13 07:26:58,197 INFO [RS-EventLoopGroup-39-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:58,197 INFO [RS-EventLoopGroup-39-2] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 43885 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:58,200 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=60497] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 07:26:58,201 INFO [RS_OPEN_META-asf911:60497-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:26:58,204 INFO [RS_OPEN_META-asf911:60497-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C60497%2C1515828415979.meta, suffix=.meta, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:26:58,211 DEBUG [RS-EventLoopGroup-40-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:26:58,211 DEBUG [RS-EventLoopGroup-40-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK] 2018-01-13 07:26:58,214 INFO [RS_OPEN_META-asf911:60497-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979/asf911.gq1.ygridcore.net%2C60497%2C1515828415979.meta.1515828418205.meta 2018-01-13 07:26:58,214 DEBUG [RS_OPEN_META-asf911:60497-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK], DatanodeInfoWithStorage[127.0.0.1:49352,DS-c601089d-c719-49d5-8041-8d8fdc76b7cc,DISK]] 2018-01-13 07:26:58,215 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:58,215 DEBUG [RS_OPEN_META-asf911:60497-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 07:26:58,215 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 07:26:58,215 INFO [RS_OPEN_META-asf911:60497-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 07:26:58,215 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 07:26:58,216 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 07:26:58,220 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:58,220 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/info 2018-01-13 07:26:58,220 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:58,221 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:58,221 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:58,224 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:58,224 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_barrier 2018-01-13 07:26:58,224 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:58,224 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:58,224 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:58,227 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:58,227 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_meta 2018-01-13 07:26:58,227 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:58,227 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:58,227 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:58,230 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:58,230 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/rep_position 2018-01-13 07:26:58,230 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:58,231 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:58,231 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:58,233 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:58,233 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/table 2018-01-13 07:26:58,233 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:58,234 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:58,234 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:58,237 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740 2018-01-13 07:26:58,239 DEBUG [RS_OPEN_META-asf911:60497-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 07:26:58,244 DEBUG [RS_OPEN_META-asf911:60497-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 07:26:58,245 INFO [RS_OPEN_META-asf911:60497-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 07:26:58,246 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 07:26:58,253 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,254 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828418253}]},"ts":1515828418253} 2018-01-13 07:26:58,254 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 07:26:58,258 DEBUG [RS_OPEN_META-asf911:60497-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,263 DEBUG [hconnection-0x1603212b-shared-pool92-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:60497 2018-01-13 07:26:58,267 INFO [RS-EventLoopGroup-39-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:58,267 INFO [RS-EventLoopGroup-39-3] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 43888 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:58,283 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 07:26:58,285 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,299 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:26:58,349 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 07:26:58,389 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 07:26:58,389 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.2830sec 2018-01-13 07:26:58,439 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 1.6590sec 2018-01-13 07:26:58,439 INFO [M:0;asf911:50890] master.HMaster(3495): hbase:meta (default replica) deployed at=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:58,449 INFO [M:0;asf911:50890] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,34281,1515828416028 belongs to an existing region server 2018-01-13 07:26:58,450 INFO [M:0;asf911:50890] master.MasterWalManager(189): Log folder hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,60497,1515828415979 belongs to an existing region server 2018-01-13 07:26:58,450 INFO [M:0;asf911:50890] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 07:26:58,452 INFO [M:0;asf911:50890] assignment.AssignmentManager(1212): Number of RegionServers=2 2018-01-13 07:26:58,452 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515828423452 2018-01-13 07:26:58,452 INFO [M:0;asf911:50890] assignment.AssignmentManager(1219): Joined the cluster in 2msec, failover=true 2018-01-13 07:26:58,457 INFO [M:0;asf911:50890] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 07:26:58,458 INFO [M:0;asf911:50890] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 07:26:58,644 DEBUG [M:0;asf911:50890] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:58,646 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:58,707 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:58,767 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:26:58,835 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741920_1098, blk_1073741921_1099, blk_1073741922_1100, blk_1073741923_1101, blk_1073741924_1102, blk_1073741925_1103, blk_1073741926_1104, blk_1073741927_1105, blk_1073741928_1106, blk_1073741929_1107, blk_1073741930_1108, blk_1073741931_1109, blk_1073741909_1087, blk_1073741910_1088, blk_1073741911_1089, blk_1073741912_1090, blk_1073741913_1091, blk_1073741914_1092, blk_1073741915_1093, blk_1073741916_1094, blk_1073741917_1095, blk_1073741918_1096, blk_1073741919_1097] 2018-01-13 07:26:58,849 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741940_1118{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:58,849 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741940_1118{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:58,860 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 07:26:58,861 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == hbase:namespace 2018-01-13 07:26:58,871 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741941_1119{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:26:58,872 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741941_1119{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:26:58,873 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:58,873 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e.: disabling compactions & flushes 2018-01-13 07:26:58,873 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:58,873 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:58,949 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:58,951 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:26:58,956 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 07:26:58,996 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:58,997 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828418996}]},"ts":1515828418996} 2018-01-13 07:26:59,001 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 07:26:59,001 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,007 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979}] 2018-01-13 07:26:59,046 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979 hbase:namespace hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,049 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979; forceNewPlan=false, retain=false 2018-01-13 07:26:59,201 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 07:26:59,206 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,206 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x57b2ff3a to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:26:59,225 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59750bbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:26:59,227 DEBUG [MultiHConnection-shared--pool93-t1] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:60497 2018-01-13 07:26:59,229 INFO [RS-EventLoopGroup-39-4] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:26:59,229 INFO [RS-EventLoopGroup-39-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 43899 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:26:59,233 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,383 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,60497,1515828415979 version=3145728 2018-01-13 07:26:59,384 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=60497] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,390 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] regionserver.HRegion(6949): Opening region: {ENCODED => f9352f3325a53760eee8eec11ba5224e, NAME => 'hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e.', STARTKEY => '', ENDKEY => ''} 2018-01-13 07:26:59,390 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace f9352f3325a53760eee8eec11ba5224e 2018-01-13 07:26:59,391 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,395 DEBUG [StoreOpener-f9352f3325a53760eee8eec11ba5224e-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/f9352f3325a53760eee8eec11ba5224e/info 2018-01-13 07:26:59,395 DEBUG [StoreOpener-f9352f3325a53760eee8eec11ba5224e-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/f9352f3325a53760eee8eec11ba5224e/info 2018-01-13 07:26:59,396 INFO [StoreOpener-f9352f3325a53760eee8eec11ba5224e-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 07:26:59,396 INFO [StoreOpener-f9352f3325a53760eee8eec11ba5224e-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:26:59,396 INFO [StoreOpener-f9352f3325a53760eee8eec11ba5224e-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:26:59,398 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/f9352f3325a53760eee8eec11ba5224e 2018-01-13 07:26:59,404 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/namespace/f9352f3325a53760eee8eec11ba5224e/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:26:59,404 INFO [RS_OPEN_PRIORITY_REGION-asf911:60497-0] regionserver.HRegion(960): Onlined f9352f3325a53760eee8eec11ba5224e; next sequenceid=2 2018-01-13 07:26:59,406 INFO [PostOpenDeployTasks:f9352f3325a53760eee8eec11ba5224e] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,414 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,414 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,415 DEBUG [PostOpenDeployTasks:f9352f3325a53760eee8eec11ba5224e] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:26:59,419 DEBUG [RS_OPEN_PRIORITY_REGION-asf911:60497-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,493 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:26:59,535 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 07:26:59,535 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:59,535 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=f9352f3325a53760eee8eec11ba5224e, target=asf911.gq1.ygridcore.net,60497,1515828415979 in 418msec 2018-01-13 07:26:59,536 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828419535}]},"ts":1515828419535} 2018-01-13 07:26:59,539 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 07:26:59,556 DEBUG [M:0;asf911:50890] zookeeper.ZKUtil(357): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 07:26:59,571 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:59,614 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 07:26:59,661 DEBUG [M:0;asf911:50890] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,703 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1570sec 2018-01-13 07:26:59,705 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,744 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,786 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,828 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,846 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:26:59,894 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 07:26:59,940 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 317msec 2018-01-13 07:27:00,090 DEBUG [M:0;asf911:50890] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,092 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,139 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,190 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,256 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,274 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:27:00,339 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:00,390 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 394msec 2018-01-13 07:27:00,412 DEBUG [M:0;asf911:50890] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 07:27:00,421 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:27:00,429 DEBUG [M:0;asf911:50890] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 07:27:00,434 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:27:00,434 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-13 07:27:00,437 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:27:00,437 INFO [M:0;asf911:50890] master.HMaster(928): Master has completed initialization 4.341sec 2018-01-13 07:27:00,438 INFO [M:0;asf911:50890] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-13 07:27:00,438 INFO [M:0;asf911:50890] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-13 07:27:00,439 INFO [M:0;asf911:50890] regionserver.HRegionServer(866): ClusterId : 57f05237-7556-4025-95c5-f7f004fd1e7d 2018-01-13 07:27:00,468 DEBUG [hconnection-0x51a18b21-shared-pool12-t15] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:27:00,469 DEBUG [Default-IPC-NioEventLoopGroup-7-1] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59697 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59697 2018-01-13 07:27:00,579 DEBUG [hconnection-0x51a18b21-shared-pool12-t15] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:60497 2018-01-13 07:27:00,581 INFO [RS-EventLoopGroup-39-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:27:00,582 INFO [RS-EventLoopGroup-39-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 43916 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:27:00,584 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:27:00,611 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0 server-side Connection retries=45 2018-01-13 07:27:00,611 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 07:27:00,611 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 07:27:00,611 INFO [Time-limited test] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 07:27:00,612 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 07:27:00,612 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 07:27:00,613 INFO [Time-limited test] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/67.195.81.155:48364 2018-01-13 07:27:00,613 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:00,613 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:00,614 INFO [Time-limited test] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:27:00,615 INFO [Time-limited test] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:27:00,616 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:48364 connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:27:00,624 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:483640x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:27:00,626 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(523): regionserver:48364-0x160ee6894500083 connected 2018-01-13 07:27:00,626 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:27:00,627 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-01-13 07:27:00,627 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=48364 2018-01-13 07:27:00,629 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=48364 2018-01-13 07:27:00,630 DEBUG [Time-limited test] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=48364 2018-01-13 07:27:00,635 INFO [RS:2;asf911:48364] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x70186d8c to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:27:00,647 DEBUG [RS:2;asf911:48364] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7edf8dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:27:00,652 INFO [RS:2;asf911:48364] regionserver.HRegionServer(866): ClusterId : 57f05237-7556-4025-95c5-f7f004fd1e7d 2018-01-13 07:27:00,652 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 07:27:00,662 DEBUG [RS:2;asf911:48364] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:27:00,663 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 07:27:00,663 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 07:27:00,671 DEBUG [RS:2;asf911:48364] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:27:00,672 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 07:27:00,672 DEBUG [RS:2;asf911:48364] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ae006fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:27:00,672 DEBUG [RS:2;asf911:48364] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:2;asf911:48364 2018-01-13 07:27:00,672 INFO [RS:2;asf911:48364] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 07:27:00,672 INFO [RS:2;asf911:48364] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 07:27:00,673 INFO [RS:2;asf911:48364] regionserver.HRegionServer(2612): reportForDuty to master=asf911.gq1.ygridcore.net,50890,1515828415770 with port=48364, startcode=1515828420610 2018-01-13 07:27:00,673 DEBUG [RS:2;asf911:48364] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:27:00,675 INFO [RS-EventLoopGroup-37-4] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.19 (auth:SIMPLE) 2018-01-13 07:27:00,675 INFO [RS-EventLoopGroup-37-4] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 54318 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:27:00,676 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,676 DEBUG [RS:2;asf911:48364] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9 2018-01-13 07:27:00,676 DEBUG [RS:2;asf911:48364] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:45316 2018-01-13 07:27:00,676 DEBUG [RS:2;asf911:48364] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 07:27:00,682 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:00,682 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:00,682 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:00,682 DEBUG [RS:2;asf911:48364] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,682 WARN [RS:2;asf911:48364] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:27:00,682 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:00,683 DEBUG [RS:2;asf911:48364] regionserver.HRegionServer(1800): logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:00,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:00,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:00,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:00,683 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,684 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:00,686 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,688 DEBUG [main-EventThread] zookeeper.ZKUtil(355): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,690 INFO [RS:2;asf911:48364] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 07:27:00,691 DEBUG [RS:2;asf911:48364] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:00,692 DEBUG [RS:2;asf911:48364] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:00,692 DEBUG [RS:2;asf911:48364] zookeeper.ZKUtil(355): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,693 INFO [RS:2;asf911:48364] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x743a1d6f to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:27:00,705 DEBUG [RS:2;asf911:48364] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fcd788b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:27:00,705 DEBUG [RS:2;asf911:48364] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 07:27:00,706 INFO [RS:2;asf911:48364] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 07:27:00,708 INFO [RS:2;asf911:48364] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-13 07:27:00,708 INFO [RS:2;asf911:48364] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 07:27:00,716 INFO [RS:2;asf911:48364] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 07:27:00,726 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-asf911:48364, corePoolSize=3, maxPoolSize=3 2018-01-13 07:27:00,726 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-asf911:48364, corePoolSize=1, maxPoolSize=1 2018-01-13 07:27:00,726 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-asf911:48364, corePoolSize=3, maxPoolSize=3 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-asf911:48364, corePoolSize=3, maxPoolSize=3 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-asf911:48364, corePoolSize=1, maxPoolSize=1 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-asf911:48364, corePoolSize=2, maxPoolSize=2 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-asf911:48364, corePoolSize=10, maxPoolSize=10 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-asf911:48364, corePoolSize=3, maxPoolSize=3 2018-01-13 07:27:00,727 DEBUG [RS:2;asf911:48364] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-asf911:48364, corePoolSize=2, maxPoolSize=2 2018-01-13 07:27:00,771 INFO [RS:2;asf911:48364] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 07:27:00,771 INFO [SplitLogWorker-asf911:48364] regionserver.SplitLogWorker(130): SplitLogWorker asf911.gq1.ygridcore.net,48364,1515828420610 starting 2018-01-13 07:27:00,793 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1535): Serving as asf911.gq1.ygridcore.net,48364,1515828420610, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:48364, sessionid=0x160ee6894500083 2018-01-13 07:27:00,793 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 07:27:00,793 DEBUG [RS:2;asf911:48364] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,793 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,48364,1515828420610' 2018-01-13 07:27:00,793 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 07:27:00,793 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:27:00,794 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 07:27:00,794 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 07:27:00,795 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 07:27:00,795 DEBUG [RS:2;asf911:48364] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:00,795 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'asf911.gq1.ygridcore.net,48364,1515828420610' 2018-01-13 07:27:00,795 DEBUG [Default-IPC-NioEventLoopGroup-7-3] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:44597 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:44597 2018-01-13 07:27:00,795 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 07:27:00,795 WARN [Time-limited test] client.ConnectionImplementation(1747): Checking master connection java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:44597 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:44597 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:44597 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2018-01-13 07:27:00,798 DEBUG [RS:2;asf911:48364] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 07:27:00,802 DEBUG [RS:2;asf911:48364] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 07:27:00,803 INFO [RS:2;asf911:48364] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 07:27:00,803 INFO [RS:2;asf911:48364] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 07:27:00,803 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:27:00,805 INFO [RS-EventLoopGroup-37-5] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:27:00,805 INFO [RS-EventLoopGroup-37-5] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 60911 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:27:00,811 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.HMaster$3(1743): Client=jenkins//67.195.81.155 create 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-13 07:27:00,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:00,985 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:01,031 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,044 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741942_1120{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 522 2018-01-13 07:27:01,044 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741942_1120 size 522 2018-01-13 07:27:01,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:01,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:01,448 DEBUG [ProcExecWrkr-14] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/.tabledesc/.tableinfo.0000000001 2018-01-13 07:27:01,451 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-11] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,452 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,453 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,453 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,453 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,453 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(6772): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry', {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:01,475 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741943_1121{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED]]} size 0 2018-01-13 07:27:01,475 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741943_1121{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|FINALIZED], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:27:01,481 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:01,482 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13.: disabling compactions & flushes 2018-01-13 07:27:01,482 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:01,488 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:01,489 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741944_1122{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:01,503 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741944_1122 size 89 2018-01-13 07:27:01,504 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741945_1123{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 89 2018-01-13 07:27:01,504 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741945_1123 size 89 2018-01-13 07:27:01,514 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:01,514 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e.: disabling compactions & flushes 2018-01-13 07:27:01,515 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:01,515 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:01,516 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741947_1125{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:01,540 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:01,540 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741948_1126{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 88 2018-01-13 07:27:01,540 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.: disabling compactions & flushes 2018-01-13 07:27:01,540 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741949_1127{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 88 2018-01-13 07:27:01,540 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:01,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741951_1129{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:01,541 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:01,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741947_1125 size 89 2018-01-13 07:27:01,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741948_1126 size 88 2018-01-13 07:27:01,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741949_1127 size 88 2018-01-13 07:27:01,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741951_1129{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:01,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741946_1124{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:01,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741946_1124{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:01,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741952_1130{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:01,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741952_1130{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:01,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741950_1128{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:01,543 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:01,544 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b.: disabling compactions & flushes 2018-01-13 07:27:01,544 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741950_1128 size 89 2018-01-13 07:27:01,544 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:01,544 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:01,545 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741953_1131{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 89 2018-01-13 07:27:01,545 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:01,545 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741953_1131 size 89 2018-01-13 07:27:01,545 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.: disabling compactions & flushes 2018-01-13 07:27:01,545 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:01,545 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:01,545 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:01,546 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d.: disabling compactions & flushes 2018-01-13 07:27:01,546 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:01,546 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:01,546 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:01,546 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187.: disabling compactions & flushes 2018-01-13 07:27:01,547 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:01,547 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:01,808 INFO [RS:2;asf911:48364] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C48364%2C1515828420610, suffix=, logDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,48364,1515828420610, archiveDir=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/oldWALs 2018-01-13 07:27:01,819 DEBUG [RS-EventLoopGroup-41-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK] 2018-01-13 07:27:01,819 DEBUG [RS-EventLoopGroup-41-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK] 2018-01-13 07:27:01,823 INFO [RS:2;asf911:48364] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/WALs/asf911.gq1.ygridcore.net,48364,1515828420610/asf911.gq1.ygridcore.net%2C48364%2C1515828420610.1515828421808 2018-01-13 07:27:01,823 DEBUG [RS:2;asf911:48364] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:49352,DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189,DISK], DatanodeInfoWithStorage[127.0.0.1:59189,DS-c2f91fd0-9011-4287-bd13-96644a3511e6,DISK]] 2018-01-13 07:27:01,905 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:01,906 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77.: disabling compactions & flushes 2018-01-13 07:27:01,906 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:01,906 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-11] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.: disabling compactions & flushes 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-11] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa.: disabling compactions & flushes 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-11] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:01,935 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-11] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:01,935 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:01,935 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:01,947 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:01,947 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347.: disabling compactions & flushes 2018-01-13 07:27:01,947 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:01,947 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,017 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:02,020 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,020 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,020 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,021 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,021 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,021 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,021 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,022 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,022 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,022 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.","families":{"info":[{"qualifier":"regioninfo","vlen":87,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,022 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347.","families":{"info":[{"qualifier":"regioninfo","vlen":88,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 07:27:02,031 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1638): Added 11 regions to meta. 2018-01-13 07:27:02,073 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:02,074 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828422073}]},"ts":1515828422073} 2018-01-13 07:27:02,077 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1786): Updated table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry state to ENABLING in META 2018-01-13 07:27:02,078 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,083 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,084 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,084 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,085 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,086 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,087 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,088 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,089 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,090 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,091 DEBUG [ProcExecWrkr-14] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,092 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979}, {pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028}, {pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028}, {pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979}, {pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610}, {pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028}, {pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028}, {pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610}, {pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979}, {pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979}, {pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610}] 2018-01-13 07:27:02,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:02,173 INFO [ProcExecWrkr-15] procedure.MasterProcedureScheduler(883): pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,173 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,174 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(883): pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,174 INFO [ProcExecWrkr-6] procedure.MasterProcedureScheduler(883): pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,174 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(883): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,175 INFO [ProcExecWrkr-16] procedure.MasterProcedureScheduler(883): pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,175 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,175 INFO [ProcExecWrkr-4] procedure.MasterProcedureScheduler(883): pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,175 INFO [ProcExecWrkr-5] procedure.MasterProcedureScheduler(883): pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,175 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(883): pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,176 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(883): pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028 testRegionAssignmentAfterMasterRecoveryDueToZKExpiry testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,176 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OFFLINE, location=asf911.gq1.ygridcore.net,48364,1515828420610; forceNewPlan=false, retain=false 2018-01-13 07:27:02,176 INFO [ProcExecWrkr-15] assignment.AssignProcedure(211): Start pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OFFLINE, location=asf911.gq1.ygridcore.net,48364,1515828420610; forceNewPlan=false, retain=false 2018-01-13 07:27:02,176 INFO [ProcExecWrkr-3] assignment.AssignProcedure(211): Start pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34281,1515828416028; forceNewPlan=false, retain=false 2018-01-13 07:27:02,176 INFO [ProcExecWrkr-6] assignment.AssignProcedure(211): Start pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-9] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-4] assignment.AssignProcedure(211): Start pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34281,1515828416028; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-16] assignment.AssignProcedure(211): Start pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-5] assignment.AssignProcedure(211): Start pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OFFLINE, location=asf911.gq1.ygridcore.net,48364,1515828420610; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34281,1515828416028; forceNewPlan=false, retain=false 2018-01-13 07:27:02,177 INFO [ProcExecWrkr-8] assignment.AssignProcedure(211): Start pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34281,1515828416028; forceNewPlan=false, retain=false 2018-01-13 07:27:02,178 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OFFLINE, location=asf911.gq1.ygridcore.net,60497,1515828415979; forceNewPlan=false, retain=false 2018-01-13 07:27:02,331 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 11 regions. 11 retained the pre-restart assignment. 2018-01-13 07:27:02,352 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=16 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,352 INFO [ProcExecWrkr-11] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,354 INFO [ProcExecWrkr-3] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,355 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,354 INFO [ProcExecWrkr-15] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,354 INFO [ProcExecWrkr-2] assignment.RegionStateStore(201): pid=15 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,354 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,352 INFO [ProcExecWrkr-13] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,352 INFO [ProcExecWrkr-12] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,355 INFO [ProcExecWrkr-4] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,355 INFO [ProcExecWrkr-6] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77., regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,361 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(208): Dispatch pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,361 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(208): Dispatch pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,361 INFO [ProcExecWrkr-11] assignment.RegionTransitionProcedure(208): Dispatch pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,361 INFO [ProcExecWrkr-13] assignment.RegionTransitionProcedure(208): Dispatch pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,367 INFO [ProcExecWrkr-3] assignment.RegionTransitionProcedure(208): Dispatch pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,367 INFO [ProcExecWrkr-14] assignment.RegionTransitionProcedure(208): Dispatch pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,370 INFO [ProcExecWrkr-2] assignment.RegionTransitionProcedure(208): Dispatch pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,370 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,367 INFO [ProcExecWrkr-12] assignment.RegionTransitionProcedure(208): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,382 INFO [ProcExecWrkr-15] assignment.RegionTransitionProcedure(208): Dispatch pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,385 INFO [ProcExecWrkr-6] assignment.RegionTransitionProcedure(208): Dispatch pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,511 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,60497,1515828415979 version=3145728 2018-01-13 07:27:02,512 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,34281,1515828416028 version=3145728 2018-01-13 07:27:02,512 DEBUG [RSProcedureDispatcher-pool89-t4] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,513 DEBUG [RSProcedureDispatcher-pool89-t4] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:34281 2018-01-13 07:27:02,513 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=60497] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,514 INFO [RS-EventLoopGroup-40-20] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:27:02,515 INFO [RS-EventLoopGroup-40-20] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 39755 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:27:02,518 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34281] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,520 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=asf911.gq1.ygridcore.net,48364,1515828420610 version=3145728 2018-01-13 07:27:02,521 DEBUG [RSProcedureDispatcher-pool89-t5] master.ServerManager(783): New admin connection to asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,521 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=60497] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,521 DEBUG [RSProcedureDispatcher-pool89-t5] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:48364 2018-01-13 07:27:02,521 DEBUG [RS_OPEN_REGION-asf911:60497-0] regionserver.HRegion(6949): Opening region: {ENCODED => 670a9ff5c823409e87537ac675a26187, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187.', STARTKEY => 'i', ENDKEY => 'j'} 2018-01-13 07:27:02,521 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=60497] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,521 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(6949): Opening region: {ENCODED => aa7fff1a9173e8d09001b7caaf5e5e8b, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b.', STARTKEY => 'h', ENDKEY => 'i'} 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:60497-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 670a9ff5c823409e87537ac675a26187 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry aa7fff1a9173e8d09001b7caaf5e5e8b 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:60497-0] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,522 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34281] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,522 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=60497] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:60497-2] regionserver.HRegion(6949): Opening region: {ENCODED => 65f465a6dfcc272619453c0eff3a0c13, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13.', STARTKEY => 'c', ENDKEY => 'd'} 2018-01-13 07:27:02,522 DEBUG [RS_OPEN_REGION-asf911:34281-0] regionserver.HRegion(6949): Opening region: {ENCODED => 0a60e0b567378965cf87f0b02cf06c77, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77.', STARTKEY => 'f', ENDKEY => 'g'} 2018-01-13 07:27:02,523 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(6949): Opening region: {ENCODED => 18f591905bbaa13514b7c6c437b3c57e, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e.', STARTKEY => 'a', ENDKEY => 'b'} 2018-01-13 07:27:02,523 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34281] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,523 DEBUG [RS_OPEN_REGION-asf911:34281-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 0a60e0b567378965cf87f0b02cf06c77 2018-01-13 07:27:02,523 DEBUG [RS_OPEN_REGION-asf911:34281-0] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,523 DEBUG [RS_OPEN_REGION-asf911:60497-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 65f465a6dfcc272619453c0eff3a0c13 2018-01-13 07:27:02,524 DEBUG [RS_OPEN_REGION-asf911:34281-2] regionserver.HRegion(6949): Opening region: {ENCODED => 7428bc7e2e038f99355fdf0180efc347, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347.', STARTKEY => 'b', ENDKEY => 'c'} 2018-01-13 07:27:02,523 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34281] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,523 INFO [RS-EventLoopGroup-41-7] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-13 07:27:02,523 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 18f591905bbaa13514b7c6c437b3c57e 2018-01-13 07:27:02,524 DEBUG [RS_OPEN_REGION-asf911:34281-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 7428bc7e2e038f99355fdf0180efc347 2018-01-13 07:27:02,524 DEBUG [RS_OPEN_REGION-asf911:60497-2] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,524 DEBUG [RS_OPEN_REGION-asf911:34281-2] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,524 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,524 INFO [RS-EventLoopGroup-41-7] ipc.ServerRpcConnection(557): Connection from 67.195.81.155 port: 58842 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "4ddfecac563384de6f087fc7fc8bc62c0362877d" user: "jenkins" date: "Sat Jan 13 07:12:53 UTC 2018" src_checksum: "d0b248ebd327a6c13d61ead0aa4930e9" version_major: 3 version_minor: 0 2018-01-13 07:27:02,528 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=48364] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,530 DEBUG [StoreOpener-670a9ff5c823409e87537ac675a26187-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/670a9ff5c823409e87537ac675a26187/info 2018-01-13 07:27:02,530 DEBUG [StoreOpener-670a9ff5c823409e87537ac675a26187-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/670a9ff5c823409e87537ac675a26187/info 2018-01-13 07:27:02,530 DEBUG [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/aa7fff1a9173e8d09001b7caaf5e5e8b/info 2018-01-13 07:27:02,530 DEBUG [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/aa7fff1a9173e8d09001b7caaf5e5e8b/info 2018-01-13 07:27:02,530 INFO [StoreOpener-670a9ff5c823409e87537ac675a26187-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,531 INFO [StoreOpener-670a9ff5c823409e87537ac675a26187-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,531 INFO [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,531 INFO [StoreOpener-670a9ff5c823409e87537ac675a26187-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,531 DEBUG [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/0a60e0b567378965cf87f0b02cf06c77/info 2018-01-13 07:27:02,531 INFO [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,531 INFO [StoreOpener-670a9ff5c823409e87537ac675a26187-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,531 INFO [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,532 INFO [StoreOpener-aa7fff1a9173e8d09001b7caaf5e5e8b-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,532 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=48364] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,533 DEBUG [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/18f591905bbaa13514b7c6c437b3c57e/info 2018-01-13 07:27:02,531 DEBUG [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/0a60e0b567378965cf87f0b02cf06c77/info 2018-01-13 07:27:02,533 DEBUG [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/18f591905bbaa13514b7c6c437b3c57e/info 2018-01-13 07:27:02,533 DEBUG [RS_OPEN_REGION-asf911:48364-0] regionserver.HRegion(6949): Opening region: {ENCODED => 5d428ce1bfa5a483bf20d1264ccf3bd7, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.', STARTKEY => 'd', ENDKEY => 'e'} 2018-01-13 07:27:02,533 DEBUG [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7428bc7e2e038f99355fdf0180efc347/info 2018-01-13 07:27:02,533 DEBUG [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7428bc7e2e038f99355fdf0180efc347/info 2018-01-13 07:27:02,532 DEBUG [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/65f465a6dfcc272619453c0eff3a0c13/info 2018-01-13 07:27:02,534 DEBUG [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/65f465a6dfcc272619453c0eff3a0c13/info 2018-01-13 07:27:02,532 INFO [RS_OPEN_REGION-asf911:60497-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,533 DEBUG [RS_OPEN_REGION-asf911:48364-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 5d428ce1bfa5a483bf20d1264ccf3bd7 2018-01-13 07:27:02,533 INFO [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,534 INFO [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,533 INFO [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,533 DEBUG [RS_OPEN_REGION-asf911:48364-1] regionserver.HRegion(6949): Opening region: {ENCODED => b9b84001e287f4bc75307879516842fa, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa.', STARTKEY => 'j', ENDKEY => ''} 2018-01-13 07:27:02,533 INFO [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,535 DEBUG [RS_OPEN_REGION-asf911:48364-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry b9b84001e287f4bc75307879516842fa 2018-01-13 07:27:02,535 DEBUG [RS_OPEN_REGION-asf911:48364-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,533 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=48364] regionserver.RSRpcServices(1907): Open testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,535 DEBUG [RS_OPEN_REGION-asf911:60497-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/670a9ff5c823409e87537ac675a26187 2018-01-13 07:27:02,535 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/aa7fff1a9173e8d09001b7caaf5e5e8b 2018-01-13 07:27:02,534 INFO [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,534 INFO [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,534 INFO [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,536 INFO [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,534 DEBUG [RS_OPEN_REGION-asf911:48364-0] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,534 INFO [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,536 INFO [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,536 INFO [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,536 INFO [StoreOpener-18f591905bbaa13514b7c6c437b3c57e-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,537 INFO [StoreOpener-65f465a6dfcc272619453c0eff3a0c13-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,536 DEBUG [RS_OPEN_REGION-asf911:48364-2] regionserver.HRegion(6949): Opening region: {ENCODED => 96f36daa2b54a675b901bda48069e68d, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d.', STARTKEY => 'g', ENDKEY => 'h'} 2018-01-13 07:27:02,536 INFO [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,537 INFO [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,537 DEBUG [RS_OPEN_REGION-asf911:48364-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 96f36daa2b54a675b901bda48069e68d 2018-01-13 07:27:02,538 INFO [StoreOpener-0a60e0b567378965cf87f0b02cf06c77-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,538 INFO [StoreOpener-7428bc7e2e038f99355fdf0180efc347-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,538 DEBUG [RS_OPEN_REGION-asf911:48364-2] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,538 INFO [RS_OPEN_REGION-asf911:60497-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,538 INFO [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,540 DEBUG [RS_OPEN_REGION-asf911:60497-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/65f465a6dfcc272619453c0eff3a0c13 2018-01-13 07:27:02,540 INFO [RS_OPEN_REGION-asf911:34281-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,540 INFO [RS_OPEN_REGION-asf911:34281-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,540 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/18f591905bbaa13514b7c6c437b3c57e 2018-01-13 07:27:02,540 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,541 DEBUG [RS_OPEN_REGION-asf911:34281-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7428bc7e2e038f99355fdf0180efc347 2018-01-13 07:27:02,541 DEBUG [RS_OPEN_REGION-asf911:34281-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/0a60e0b567378965cf87f0b02cf06c77 2018-01-13 07:27:02,541 DEBUG [StoreOpener-b9b84001e287f4bc75307879516842fa-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b9b84001e287f4bc75307879516842fa/info 2018-01-13 07:27:02,541 DEBUG [StoreOpener-b9b84001e287f4bc75307879516842fa-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b9b84001e287f4bc75307879516842fa/info 2018-01-13 07:27:02,542 INFO [StoreOpener-b9b84001e287f4bc75307879516842fa-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,542 INFO [StoreOpener-b9b84001e287f4bc75307879516842fa-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,542 DEBUG [RS_OPEN_REGION-asf911:60497-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,542 INFO [StoreOpener-b9b84001e287f4bc75307879516842fa-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,543 INFO [StoreOpener-b9b84001e287f4bc75307879516842fa-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,543 DEBUG [RS_OPEN_REGION-asf911:34281-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,544 DEBUG [RS_OPEN_REGION-asf911:60497-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,544 DEBUG [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/5d428ce1bfa5a483bf20d1264ccf3bd7/info 2018-01-13 07:27:02,545 INFO [RS_OPEN_REGION-asf911:48364-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,545 DEBUG [RS_OPEN_REGION-asf911:60497-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/aa7fff1a9173e8d09001b7caaf5e5e8b/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,544 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,545 DEBUG [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/96f36daa2b54a675b901bda48069e68d/info 2018-01-13 07:27:02,545 INFO [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(960): Onlined aa7fff1a9173e8d09001b7caaf5e5e8b; next sequenceid=2 2018-01-13 07:27:02,545 DEBUG [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/5d428ce1bfa5a483bf20d1264ccf3bd7/info 2018-01-13 07:27:02,545 DEBUG [RS_OPEN_REGION-asf911:34281-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,546 INFO [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,546 INFO [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,545 DEBUG [RS_OPEN_REGION-asf911:48364-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b9b84001e287f4bc75307879516842fa 2018-01-13 07:27:02,545 DEBUG [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/96f36daa2b54a675b901bda48069e68d/info 2018-01-13 07:27:02,546 INFO [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,547 INFO [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,547 INFO [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,547 INFO [StoreOpener-5d428ce1bfa5a483bf20d1264ccf3bd7-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,547 INFO [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,552 INFO [PostOpenDeployTasks:aa7fff1a9173e8d09001b7caaf5e5e8b] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,552 INFO [StoreOpener-96f36daa2b54a675b901bda48069e68d-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,553 DEBUG [RS_OPEN_REGION-asf911:34281-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/7428bc7e2e038f99355fdf0180efc347/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,553 INFO [RS_OPEN_REGION-asf911:34281-2] regionserver.HRegion(960): Onlined 7428bc7e2e038f99355fdf0180efc347; next sequenceid=2 2018-01-13 07:27:02,553 DEBUG [RS_OPEN_REGION-asf911:60497-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/65f465a6dfcc272619453c0eff3a0c13/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,553 INFO [RS_OPEN_REGION-asf911:60497-2] regionserver.HRegion(960): Onlined 65f465a6dfcc272619453c0eff3a0c13; next sequenceid=2 2018-01-13 07:27:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,553 INFO [RS_OPEN_REGION-asf911:48364-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,553 INFO [RS_OPEN_REGION-asf911:48364-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,554 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=16 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,554 DEBUG [RS_OPEN_REGION-asf911:48364-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/5d428ce1bfa5a483bf20d1264ccf3bd7 2018-01-13 07:27:02,554 DEBUG [RS_OPEN_REGION-asf911:48364-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/96f36daa2b54a675b901bda48069e68d 2018-01-13 07:27:02,554 DEBUG [RS_OPEN_REGION-asf911:48364-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,555 DEBUG [RS_OPEN_REGION-asf911:34281-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/0a60e0b567378965cf87f0b02cf06c77/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,554 DEBUG [RS_OPEN_REGION-asf911:34281-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/18f591905bbaa13514b7c6c437b3c57e/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,554 DEBUG [PostOpenDeployTasks:aa7fff1a9173e8d09001b7caaf5e5e8b] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:02,555 INFO [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(960): Onlined 18f591905bbaa13514b7c6c437b3c57e; next sequenceid=2 2018-01-13 07:27:02,557 DEBUG [RS_OPEN_REGION-asf911:60497-1] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,555 INFO [RS_OPEN_REGION-asf911:34281-0] regionserver.HRegion(960): Onlined 0a60e0b567378965cf87f0b02cf06c77; next sequenceid=2 2018-01-13 07:27:02,555 INFO [PostOpenDeployTasks:65f465a6dfcc272619453c0eff3a0c13] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,558 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(6949): Opening region: {ENCODED => 6cf6a059941c0ba105d47976a87fccb1, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.', STARTKEY => '', ENDKEY => 'a'} 2018-01-13 07:27:02,555 DEBUG [RS_OPEN_REGION-asf911:60497-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/670a9ff5c823409e87537ac675a26187/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,558 INFO [RS_OPEN_REGION-asf911:60497-0] regionserver.HRegion(960): Onlined 670a9ff5c823409e87537ac675a26187; next sequenceid=2 2018-01-13 07:27:02,558 INFO [PostOpenDeployTasks:7428bc7e2e038f99355fdf0180efc347] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,558 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 6cf6a059941c0ba105d47976a87fccb1 2018-01-13 07:27:02,562 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,563 DEBUG [RS_OPEN_REGION-asf911:48364-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,563 INFO [PostOpenDeployTasks:18f591905bbaa13514b7c6c437b3c57e] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,563 INFO [ProcExecWrkr-5] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,563 INFO [PostOpenDeployTasks:670a9ff5c823409e87537ac675a26187] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,564 DEBUG [PostOpenDeployTasks:7428bc7e2e038f99355fdf0180efc347] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,563 INFO [PostOpenDeployTasks:0a60e0b567378965cf87f0b02cf06c77] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,567 INFO [ProcExecWrkr-7] assignment.RegionStateStore(201): pid=11 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,563 DEBUG [RS_OPEN_REGION-asf911:48364-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,567 DEBUG [PostOpenDeployTasks:18f591905bbaa13514b7c6c437b3c57e] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:02,567 INFO [ProcExecWrkr-8] assignment.RegionStateStore(201): pid=9 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,567 DEBUG [PostOpenDeployTasks:65f465a6dfcc272619453c0eff3a0c13] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,572 DEBUG [RS_OPEN_REGION-asf911:34281-1] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. on asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,570 DEBUG [RS_OPEN_REGION-asf911:34281-2] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. on asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,574 DEBUG [RS_OPEN_REGION-asf911:60497-2] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,575 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,575 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(6949): Opening region: {ENCODED => 63847c70149ab9a684442e6301bf4c16, NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.', STARTKEY => 'e', ENDKEY => 'f'} 2018-01-13 07:27:02,575 INFO [ProcExecWrkr-1] assignment.RegionStateStore(201): pid=17 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,575 DEBUG [PostOpenDeployTasks:0a60e0b567378965cf87f0b02cf06c77] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:02,575 DEBUG [PostOpenDeployTasks:670a9ff5c823409e87537ac675a26187] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:02,578 DEBUG [RS_OPEN_REGION-asf911:34281-0] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. on asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,580 DEBUG [RS_OPEN_REGION-asf911:60497-0] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,584 DEBUG [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6cf6a059941c0ba105d47976a87fccb1/info 2018-01-13 07:27:02,584 DEBUG [RS_OPEN_REGION-asf911:48364-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b9b84001e287f4bc75307879516842fa/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,584 INFO [RS_OPEN_REGION-asf911:48364-1] regionserver.HRegion(960): Onlined b9b84001e287f4bc75307879516842fa; next sequenceid=2 2018-01-13 07:27:02,580 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 63847c70149ab9a684442e6301bf4c16 2018-01-13 07:27:02,585 DEBUG [RS_OPEN_REGION-asf911:48364-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/96f36daa2b54a675b901bda48069e68d/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,585 INFO [RS_OPEN_REGION-asf911:48364-2] regionserver.HRegion(960): Onlined 96f36daa2b54a675b901bda48069e68d; next sequenceid=2 2018-01-13 07:27:02,584 DEBUG [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6cf6a059941c0ba105d47976a87fccb1/info 2018-01-13 07:27:02,585 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(807): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,586 INFO [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,586 DEBUG [RS_OPEN_REGION-asf911:48364-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/5d428ce1bfa5a483bf20d1264ccf3bd7/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,586 INFO [RS_OPEN_REGION-asf911:48364-0] regionserver.HRegion(960): Onlined 5d428ce1bfa5a483bf20d1264ccf3bd7; next sequenceid=2 2018-01-13 07:27:02,586 INFO [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,587 INFO [PostOpenDeployTasks:b9b84001e287f4bc75307879516842fa] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,587 INFO [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,592 INFO [StoreOpener-6cf6a059941c0ba105d47976a87fccb1-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,592 INFO [PostOpenDeployTasks:96f36daa2b54a675b901bda48069e68d] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,593 INFO [PostOpenDeployTasks:5d428ce1bfa5a483bf20d1264ccf3bd7] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,593 INFO [ProcExecWrkr-11] assignment.RegionStateStore(201): pid=18 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,593 DEBUG [PostOpenDeployTasks:b9b84001e287f4bc75307879516842fa] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610; rit=OPENING, location=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,593 INFO [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,594 INFO [ProcExecWrkr-13] assignment.RegionStateStore(201): pid=15 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,594 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,594 DEBUG [PostOpenDeployTasks:96f36daa2b54a675b901bda48069e68d] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:02,599 DEBUG [RS_OPEN_REGION-asf911:48364-1] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. on asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,599 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6cf6a059941c0ba105d47976a87fccb1 2018-01-13 07:27:02,598 DEBUG [PostOpenDeployTasks:5d428ce1bfa5a483bf20d1264ccf3bd7] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:02,608 DEBUG [RS_OPEN_REGION-asf911:48364-2] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. on asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,608 DEBUG [RS_OPEN_REGION-asf911:48364-0] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. on asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,611 DEBUG [RS_OPEN_REGION-asf911:60497-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,617 DEBUG [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/63847c70149ab9a684442e6301bf4c16/info 2018-01-13 07:27:02,617 DEBUG [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/63847c70149ab9a684442e6301bf4c16/info 2018-01-13 07:27:02,617 INFO [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-13 07:27:02,617 INFO [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-13 07:27:02,618 INFO [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=781648, freeSize=1043180656, maxSize=1043962304, heapSize=781648, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 07:27:02,618 INFO [StoreOpener-63847c70149ab9a684442e6301bf4c16-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 07:27:02,618 DEBUG [RS_OPEN_REGION-asf911:60497-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/6cf6a059941c0ba105d47976a87fccb1/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,619 INFO [RS_OPEN_REGION-asf911:60497-1] regionserver.HRegion(960): Onlined 6cf6a059941c0ba105d47976a87fccb1; next sequenceid=2 2018-01-13 07:27:02,619 INFO [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,620 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/63847c70149ab9a684442e6301bf4c16 2018-01-13 07:27:02,620 INFO [PostOpenDeployTasks:6cf6a059941c0ba105d47976a87fccb1] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979; rit=OPENING, location=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,622 INFO [ProcExecWrkr-3] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,622 DEBUG [PostOpenDeployTasks:6cf6a059941c0ba105d47976a87fccb1] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:02,622 DEBUG [RS_OPEN_REGION-asf911:34281-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-13 07:27:02,622 DEBUG [RS_OPEN_REGION-asf911:60497-1] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. on asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,626 DEBUG [RS_OPEN_REGION-asf911:34281-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/default/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/63847c70149ab9a684442e6301bf4c16/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 07:27:02,626 INFO [RS_OPEN_REGION-asf911:34281-1] regionserver.HRegion(960): Onlined 63847c70149ab9a684442e6301bf4c16; next sequenceid=2 2018-01-13 07:27:02,627 INFO [PostOpenDeployTasks:63847c70149ab9a684442e6301bf4c16] regionserver.HRegionServer(2197): Post open deploy tasks for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028; rit=OPENING, location=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,629 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16., regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,629 DEBUG [PostOpenDeployTasks:63847c70149ab9a684442e6301bf4c16] regionserver.HRegionServer(2221): Finished post open deploy task for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:02,629 DEBUG [RS_OPEN_REGION-asf911:34281-1] handler.OpenRegionHandler(128): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. on asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=16, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=9, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=17, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=11, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=17, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=670a9ff5c823409e87537ac675a26187, target=asf911.gq1.ygridcore.net,60497,1515828415979 in 501msec 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1250): Finished pid=9, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=18f591905bbaa13514b7c6c437b3c57e, target=asf911.gq1.ygridcore.net,34281,1515828416028 in 493msec 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=14, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=16, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=aa7fff1a9173e8d09001b7caaf5e5e8b, target=asf911.gq1.ygridcore.net,60497,1515828415979 in 483msec 2018-01-13 07:27:02,647 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,648 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=14, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=0a60e0b567378965cf87f0b02cf06c77, target=asf911.gq1.ygridcore.net,34281,1515828416028 in 493msec 2018-01-13 07:27:02,648 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1250): Finished pid=11, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=65f465a6dfcc272619453c0eff3a0c13, target=asf911.gq1.ygridcore.net,60497,1515828415979 in 493msec 2018-01-13 07:27:02,648 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=7428bc7e2e038f99355fdf0180efc347, target=asf911.gq1.ygridcore.net,34281,1515828416028 in 493msec 2018-01-13 07:27:02,733 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=18, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=15, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=12, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=15, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=96f36daa2b54a675b901bda48069e68d, target=asf911.gq1.ygridcore.net,48364,1515828420610 in 526msec 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=18, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=b9b84001e287f4bc75307879516842fa, target=asf911.gq1.ygridcore.net,48364,1515828420610 in 526msec 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=13, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:02,828 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1250): Finished pid=13, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=63847c70149ab9a684442e6301bf4c16, target=asf911.gq1.ygridcore.net,34281,1515828416028 in 540msec 2018-01-13 07:27:02,827 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1250): Finished pid=12, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=5d428ce1bfa5a483bf20d1264ccf3bd7, target=asf911.gq1.ygridcore.net,48364,1515828420610 in 526msec 2018-01-13 07:27:02,876 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:27:02,876 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:27:02,877 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-13 07:27:02,884 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; resume parent processing. 2018-01-13 07:27:02,884 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:02,884 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry, region=6cf6a059941c0ba105d47976a87fccb1, target=asf911.gq1.ygridcore.net,60497,1515828415979 in 533msec 2018-01-13 07:27:02,885 DEBUG [ProcExecWrkr-2] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testRegionAssignmentAfterMasterRecoveryDueToZKExpiry","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515828422885}]},"ts":1515828422885} 2018-01-13 07:27:02,889 INFO [ProcExecWrkr-2] hbase.MetaTableAccessor(1786): Updated table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry state to ENABLED in META 2018-01-13 07:27:02,986 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:03,065 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry in 2.1750sec 2018-01-13 07:27:03,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] master.MasterRpcServices(1157): Checking to see if procedure is done pid=7 2018-01-13 07:27:03,099 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3638): Operation: CREATE, Table Name: default:testRegionAssignmentAfterMasterRecoveryDueToZKExpiry completed 2018-01-13 07:27:03,100 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-13 07:27:03,112 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,112 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,113 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:27:03,113 ERROR [Time-limited test] helpers.MarkerIgnoringBase(159): Test recovery from zk session expired org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,112 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,112 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,112 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,114 WARN [main-EventThread] zookeeper.ZKUtil(361): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,114 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,114 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,113 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,113 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,113 INFO [Time-limited test] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,50890,1515828415770' ***** 2018-01-13 07:27:03,115 INFO [Time-limited test] regionserver.HRegionServer(2172): STOPPED: Stopped by Time-limited test 2018-01-13 07:27:03,114 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:27:03,115 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): Unexpected exception handling nodeDeleted event org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,114 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,115 WARN [main-EventThread] zookeeper.ZKUtil(618): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,115 ERROR [main-EventThread] zookeeper.ZKWatcher(578): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,116 ERROR [main-EventThread] master.ActiveMasterManager(280): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:221) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,115 DEBUG [M:0;asf911:50890] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4403e5c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-01-13 07:27:03,115 WARN [Time-limited test] zookeeper.ZKUtil(618): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,116 ERROR [Time-limited test] zookeeper.ZKWatcher(578): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,116 ERROR [Time-limited test] master.ActiveMasterManager(280): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.master.HMaster.abort(HMaster.java:2616) at org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:361) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,116 INFO [M:0;asf911:50890] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,50890,1515828415770 2018-01-13 07:27:03,117 DEBUG [M:0;asf911:50890] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:27:03,115 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,117 DEBUG [main-EventThread] master.ActiveMasterManager(132): No master available. Notifying waiting threads 2018-01-13 07:27:03,117 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,60497,1515828415979: killAll ***** 2018-01-13 07:27:03,117 INFO [M:0;asf911:50890] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5e1995ef to localhost:59094 2018-01-13 07:27:03,118 DEBUG [M:0;asf911:50890] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,117 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:27:03,118 INFO [M:0;asf911:50890] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,50890,1515828415770; all regions closed. 2018-01-13 07:27:03,118 DEBUG [M:0;asf911:50890] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,118 INFO [M:0;asf911:50890] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,50890,1515828415770 had [[ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,50890,1515828415770-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:50890 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,50890,1515828415770-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,50890,1515828415770-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,50890,1515828415770-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,50890,1515828415770-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:27:03,118 INFO [M:0;asf911:50890] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-13 07:27:03,121 INFO [M:0;asf911:50890] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-13 07:27:03,123 DEBUG [M:0;asf911:50890] master.HMaster(1187): Stopping service threads 2018-01-13 07:27:03,125 WARN [M:0;asf911:50890] zookeeper.ZKUtil(618): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,125 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,125 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,125 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,125 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,125 ERROR [M:0;asf911:50890] zookeeper.ZKWatcher(578): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,126 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1350041600, "init" : 792723456, "max" : 2609905664, "used" : 529097240 }, "NonHeapMemoryUsage" : { "committed" : 141754368, "init" : 2555904, "max" : -1, "used" : 139738600 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 90430, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 365, "TotalCallTime_min" : 0, "TotalCallTime_max" : 15, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 15, "TotalCallTime_99th_percentile" : 15, "TotalCallTime_99.9th_percentile" : 15, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_3-10" : 5, "TotalCallTime_TimeRangeCount_10-30" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 323, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 365, "ResponseSize_min" : 0, "ResponseSize_max" : 6712, "ResponseSize_mean" : 562, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 51, "ResponseSize_75th_percentile" : 766, "ResponseSize_90th_percentile" : 4317, "ResponseSize_95th_percentile" : 5514, "ResponseSize_98th_percentile" : 6233, "ResponseSize_99th_percentile" : 6472, "ResponseSize_99.9th_percentile" : 6688, "ResponseSize_SizeRangeCount_0-10" : 12, "ResponseSize_SizeRangeCount_10-100" : 17, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_1000-10000" : 10, "ResponseSize_SizeRangeCount_100000000-inf" : 323, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 365, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 15, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 15, "ProcessCallTime_99th_percentile" : 15, "ProcessCallTime_99.9th_percentile" : 15, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 10, "ProcessCallTime_TimeRangeCount_3-10" : 5, "ProcessCallTime_TimeRangeCount_10-30" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 58, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 365, "RequestSize_min" : 78, "RequestSize_max" : 986, "RequestSize_mean" : 186, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 126, "RequestSize_75th_percentile" : 148, "RequestSize_90th_percentile" : 161, "RequestSize_95th_percentile" : 699, "RequestSize_98th_percentile" : 983, "RequestSize_99th_percentile" : 984, "RequestSize_99.9th_percentile" : 985, "RequestSize_SizeRangeCount_10-100" : 14, "RequestSize_SizeRangeCount_100-1000" : 28, "RequestSize_SizeRangeCount_100000000-inf" : 323, "sentBytes" : 59459, "QueueCallTime_num_ops" : 365, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 42, "QueueCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043180656, "blockCacheCount" : 5, "blockCacheSize" : 781648, "blockCacheHitCount" : 25, "blockCacheHitCountPrimary" : 25, "blockCacheMissCount" : 5, "blockCacheMissCountPrimary" : 5, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 83.33333333333334, "blockCacheExpressHitPercent" : 83.33333333333334, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 5, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 25, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 25, "l1CacheMissCount" : 5, "l1CacheHitRatio" : 0.8333333333333334, "l1CacheMissRatio" : 0.16666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:27:03,126 ERROR [M:0;asf911:50890] master.ActiveMasterManager(280): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1135) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,128 INFO [M:0;asf911:50890] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:27:03,128 INFO [M:0;asf911:50890] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x57b2ff3a to localhost:59094 2018-01-13 07:27:03,128 DEBUG [M:0;asf911:50890] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,129 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,60497,1515828415979 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,60497,1515828415979: killAll ***** 2018-01-13 07:27:03,129 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,60497,1515828415979' ***** 2018-01-13 07:27:03,130 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:27:03,130 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,34281,1515828416028: killAll ***** 2018-01-13 07:27:03,130 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:27:03,130 INFO [RS:0;asf911:60497] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:27:03,131 INFO [RS:0;asf911:60497] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:27:03,131 INFO [SplitLogWorker-asf911:60497] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:27:03,131 INFO [M:0;asf911:50890] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:27:03,133 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:27:03,131 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:27:03,131 INFO [RS:0;asf911:60497] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:27:03,136 INFO [M:0;asf911:50890] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:27:03,136 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:27:03,131 INFO [SplitLogWorker-asf911:60497] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,60497,1515828415979 exiting 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,136 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,136 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,136 INFO [M:0;asf911:50890] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:27:03,136 INFO [RS:0;asf911:60497] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,140 DEBUG [RS_CLOSE_REGION-asf911:60497-0] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,139 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:03,142 DEBUG [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b.: disabling compactions & flushes 2018-01-13 07:27:03,143 DEBUG [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:03,140 DEBUG [RS_CLOSE_REGION-asf911:60497-2] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:03,140 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,140 DEBUG [RS_CLOSE_REGION-asf911:60497-1] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:27:03,140 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1350041600, "init" : 792723456, "max" : 2609905664, "used" : 529748856 }, "NonHeapMemoryUsage" : { "committed" : 141754368, "init" : 2555904, "max" : -1, "used" : 139745512 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 90430, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 365, "TotalCallTime_min" : 0, "TotalCallTime_max" : 15, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 15, "TotalCallTime_99th_percentile" : 15, "TotalCallTime_99.9th_percentile" : 15, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_3-10" : 5, "TotalCallTime_TimeRangeCount_10-30" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 323, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 365, "ResponseSize_min" : 0, "ResponseSize_max" : 6712, "ResponseSize_mean" : 562, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 51, "ResponseSize_75th_percentile" : 766, "ResponseSize_90th_percentile" : 4317, "ResponseSize_95th_percentile" : 5514, "ResponseSize_98th_percentile" : 6233, "ResponseSize_99th_percentile" : 6472, "ResponseSize_99.9th_percentile" : 6688, "ResponseSize_SizeRangeCount_0-10" : 12, "ResponseSize_SizeRangeCount_10-100" : 17, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_1000-10000" : 10, "ResponseSize_SizeRangeCount_100000000-inf" : 323, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 365, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 15, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 15, "ProcessCallTime_99th_percentile" : 15, "ProcessCallTime_99.9th_percentile" : 15, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 10, "ProcessCallTime_TimeRangeCount_3-10" : 5, "ProcessCallTime_TimeRangeCount_10-30" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 58, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 365, "RequestSize_min" : 78, "RequestSize_max" : 986, "RequestSize_mean" : 186, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 126, "RequestSize_75th_percentile" : 148, "RequestSize_90th_percentile" : 161, "RequestSize_95th_percentile" : 699, "RequestSize_98th_percentile" : 983, "RequestSize_99th_percentile" : 984, "RequestSize_99.9th_percentile" : 985, "RequestSize_SizeRangeCount_10-100" : 14, "RequestSize_SizeRangeCount_100-1000" : 28, "RequestSize_SizeRangeCount_100000000-inf" : 323, "sentBytes" : 59459, "QueueCallTime_num_ops" : 365, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 42, "QueueCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043180656, "blockCacheCount" : 5, "blockCacheSize" : 781648, "blockCacheHitCount" : 25, "blockCacheHitCountPrimary" : 25, "blockCacheMissCount" : 5, "blockCacheMissCountPrimary" : 5, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 83.33333333333334, "blockCacheExpressHitPercent" : 83.33333333333334, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 5, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 25, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 25, "l1CacheMissCount" : 5, "l1CacheHitRatio" : 0.8333333333333334, "l1CacheMissRatio" : 0.16666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:27:03,146 DEBUG [RS_CLOSE_REGION-asf911:60497-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e.: disabling compactions & flushes 2018-01-13 07:27:03,145 DEBUG [RS:0;asf911:60497] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:27:03,147 INFO [RS:0;asf911:60497] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x5840413c to localhost:59094 2018-01-13 07:27:03,150 DEBUG [RS:0;asf911:60497] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,145 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,145 DEBUG [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13.: disabling compactions & flushes 2018-01-13 07:27:03,150 DEBUG [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:03,150 INFO [RS:0;asf911:60497] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:27:03,150 INFO [RS:0;asf911:60497] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:27:03,151 INFO [RS:0;asf911:60497] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:27:03,147 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741936_1114{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:27:03,147 DEBUG [RS_CLOSE_REGION-asf911:60497-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:27:03,147 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 07:27:03,151 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 07:27:03,147 INFO [M:1;asf911:36180] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,36180,1515828415912 from backup master directory 2018-01-13 07:27:03,151 INFO [StoreCloserThread-hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,151 DEBUG [RS_CLOSE_META-asf911:60497-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 07:27:03,151 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741936_1114{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 214 2018-01-13 07:27:03,151 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1388): Waiting on 6 regions to close 2018-01-13 07:27:03,151 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,150 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=50890] master.MasterRpcServices(503): Region server asf911.gq1.ygridcore.net,34281,1515828416028 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,34281,1515828416028: killAll ***** 2018-01-13 07:27:03,150 INFO [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:03,154 INFO [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:03,154 DEBUG [RS:0;asf911:60497] regionserver.HRegionServer(1392): Online Regions={aa7fff1a9173e8d09001b7caaf5e5e8b=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b., f9352f3325a53760eee8eec11ba5224e=hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e., 65f465a6dfcc272619453c0eff3a0c13=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13., 1588230740=hbase:meta,,1.1588230740, 670a9ff5c823409e87537ac675a26187=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187., 6cf6a059941c0ba105d47976a87fccb1=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.} 2018-01-13 07:27:03,154 DEBUG [RS_CLOSE_META-asf911:60497-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 07:27:03,155 DEBUG [RS_CLOSE_META-asf911:60497-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 07:27:03,152 ERROR [RS_CLOSE_REGION-asf911:60497-1] regionserver.HRegion(1620): Memstore size is 78 2018-01-13 07:27:03,155 INFO [M:0;asf911:50890] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,50890,1515828415770_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:27:03,155 DEBUG [RS_CLOSE_REGION-asf911:60497-2] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1515828420809.65f465a6dfcc272619453c0eff3a0c13. 2018-01-13 07:27:03,155 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,34281,1515828416028' ***** 2018-01-13 07:27:03,156 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:27:03,154 DEBUG [RS_CLOSE_REGION-asf911:60497-0] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1515828420809.aa7fff1a9173e8d09001b7caaf5e5e8b. 2018-01-13 07:27:03,156 ERROR [main] helpers.MarkerIgnoringBase(143): ***** ABORTING region server asf911.gq1.ygridcore.net,48364,1515828420610: killAll ***** 2018-01-13 07:27:03,156 INFO [RS:1;asf911:34281] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:27:03,155 DEBUG [RS_CLOSE_REGION-asf911:60497-2] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:03,155 INFO [M:0;asf911:50890] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:27:03,157 INFO [SplitLogWorker-asf911:34281] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:27:03,155 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,155 INFO [RS_CLOSE_REGION-asf911:60497-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:27:03,159 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 07:27:03,159 INFO [SplitLogWorker-asf911:34281] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,34281,1515828416028 exiting 2018-01-13 07:27:03,159 INFO [M:0;asf911:50890] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:27:03,159 DEBUG [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187.: disabling compactions & flushes 2018-01-13 07:27:03,160 DEBUG [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:03,156 INFO [RS:1;asf911:34281] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:27:03,156 ERROR [main] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 07:27:03,161 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,161 INFO [RS_CLOSE_REGION-asf911:60497-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:03,161 DEBUG [RS_CLOSE_REGION-asf911:60497-2] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1515828420809.670a9ff5c823409e87537ac675a26187. 2018-01-13 07:27:03,156 DEBUG [RS_CLOSE_REGION-asf911:60497-0] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:03,165 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:27:03,160 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:27:03,160 INFO [RS:1;asf911:34281] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:27:03,168 INFO [RS:1;asf911:34281] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:27:03,168 WARN [M:1;asf911:36180] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 07:27:03,160 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 07:27:03,160 DEBUG [RS_CLOSE_REGION-asf911:60497-1] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515828418458.f9352f3325a53760eee8eec11ba5224e. 2018-01-13 07:27:03,168 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 07:27:03,168 INFO [M:1;asf911:36180] master.ActiveMasterManager(181): Registered Active Master=asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:27:03,168 INFO [main] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1350041600, "init" : 792723456, "max" : 2609905664, "used" : 532226448 }, "NonHeapMemoryUsage" : { "committed" : 141754368, "init" : 2555904, "max" : -1, "used" : 139796712 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 90430, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 365, "TotalCallTime_min" : 0, "TotalCallTime_max" : 15, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 1, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 9, "TotalCallTime_95th_percentile" : 14, "TotalCallTime_98th_percentile" : 15, "TotalCallTime_99th_percentile" : 15, "TotalCallTime_99.9th_percentile" : 15, "TotalCallTime_TimeRangeCount_0-1" : 24, "TotalCallTime_TimeRangeCount_1-3" : 10, "TotalCallTime_TimeRangeCount_3-10" : 5, "TotalCallTime_TimeRangeCount_10-30" : 3, "TotalCallTime_TimeRangeCount_600000-inf" : 323, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 365, "ResponseSize_min" : 0, "ResponseSize_max" : 6712, "ResponseSize_mean" : 562, "ResponseSize_25th_percentile" : 10, "ResponseSize_median" : 51, "ResponseSize_75th_percentile" : 766, "ResponseSize_90th_percentile" : 4317, "ResponseSize_95th_percentile" : 5514, "ResponseSize_98th_percentile" : 6233, "ResponseSize_99th_percentile" : 6472, "ResponseSize_99.9th_percentile" : 6688, "ResponseSize_SizeRangeCount_0-10" : 12, "ResponseSize_SizeRangeCount_10-100" : 17, "ResponseSize_SizeRangeCount_100-1000" : 3, "ResponseSize_SizeRangeCount_1000-10000" : 10, "ResponseSize_SizeRangeCount_100000000-inf" : 323, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 3, "ProcessCallTime_num_ops" : 365, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 15, "ProcessCallTime_mean" : 2, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 1, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 9, "ProcessCallTime_95th_percentile" : 14, "ProcessCallTime_98th_percentile" : 15, "ProcessCallTime_99th_percentile" : 15, "ProcessCallTime_99.9th_percentile" : 15, "ProcessCallTime_TimeRangeCount_0-1" : 24, "ProcessCallTime_TimeRangeCount_1-3" : 10, "ProcessCallTime_TimeRangeCount_3-10" : 5, "ProcessCallTime_TimeRangeCount_10-30" : 3, "ProcessCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 58, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 365, "RequestSize_min" : 78, "RequestSize_max" : 986, "RequestSize_mean" : 186, "RequestSize_25th_percentile" : 85, "RequestSize_median" : 126, "RequestSize_75th_percentile" : 148, "RequestSize_90th_percentile" : 161, "RequestSize_95th_percentile" : 699, "RequestSize_98th_percentile" : 983, "RequestSize_99th_percentile" : 984, "RequestSize_99.9th_percentile" : 985, "RequestSize_SizeRangeCount_10-100" : 14, "RequestSize_SizeRangeCount_100-1000" : 28, "RequestSize_SizeRangeCount_100000000-inf" : 323, "sentBytes" : 59459, "QueueCallTime_num_ops" : 365, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 42, "QueueCallTime_TimeRangeCount_600000-inf" : 323, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:59094", "tag.serverName" : "asf911.gq1.ygridcore.net,57040,1515828329910", "tag.clusterId" : "b696352d-1ffd-427e-9e42-770be24a4b2e", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 0, "storeCount" : 0, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1515828329910, "averageRegionSize" : 0, "totalRequestCount" : 0, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 0, "rpcScanRequestCount" : 0, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 0, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043180656, "blockCacheCount" : 5, "blockCacheSize" : 781648, "blockCacheHitCount" : 25, "blockCacheHitCountPrimary" : 25, "blockCacheMissCount" : 5, "blockCacheMissCountPrimary" : 5, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 83.33333333333334, "blockCacheExpressHitPercent" : 83.33333333333334, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 5, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 25, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 25, "l1CacheMissCount" : 5, "l1CacheHitRatio" : 0.8333333333333334, "l1CacheMissRatio" : 0.16666666666666666, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 0, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 0, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 0, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 0, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-01-13 07:27:03,168 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:27:03,167 DEBUG [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.: disabling compactions & flushes 2018-01-13 07:27:03,165 WARN [M:0;asf911:50890] regionserver.HRegionServer(1146): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf911.gq1.ygridcore.net,50890,1515828415770 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1591) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1143) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:564) at java.lang.Thread.run(Thread.java:748) 2018-01-13 07:27:03,171 DEBUG [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:03,171 DEBUG [main] ipc.NettyRpcConnection(255): Connecting to asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:27:03,169 DEBUG [RS_CLOSE_REGION-asf911:34281-2] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:03,169 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,176 DEBUG [RS:1;asf911:34281] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:27:03,169 DEBUG [RS_CLOSE_REGION-asf911:34281-1] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:03,168 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 07:27:03,168 DEBUG [RS_CLOSE_REGION-asf911:34281-0] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:03,178 ERROR [RS_CLOSE_META-asf911:60497-0] regionserver.HRegion(1620): Memstore size is 20347 2018-01-13 07:27:03,180 WARN [main] regionserver.HRegionServer(2402): Unable to report fatal error to master org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:50890 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:50890 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRSFatalError(RegionServerStatusProtos.java:13568) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2399) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abortRegionServer(MiniHBaseCluster.java:192) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$200(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$2.run(MiniHBaseCluster.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.abort(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2412) at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:824) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) Caused by: java.net.ConnectException: Call to asf911.gq1.ygridcore.net/67.195.81.155:50890 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:50890 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:659) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:678) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:50890 at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused ... 1 more 2018-01-13 07:27:03,185 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,48364,1515828420610' ***** 2018-01-13 07:27:03,185 INFO [main] regionserver.HRegionServer(2172): STOPPED: killAll 2018-01-13 07:27:03,177 DEBUG [RS_CLOSE_REGION-asf911:34281-1] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77.: disabling compactions & flushes 2018-01-13 07:27:03,176 INFO [RS:1;asf911:34281] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x6d6c3f5f to localhost:59094 2018-01-13 07:27:03,175 DEBUG [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e.: disabling compactions & flushes 2018-01-13 07:27:03,185 DEBUG [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:03,174 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,173 INFO [M:0;asf911:50890] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,50890,1515828415770; zookeeper connection closed. 2018-01-13 07:27:03,186 INFO [M:0;asf911:50890] regionserver.HRegionServer(1157): M:0;asf911:50890 exiting 2018-01-13 07:27:03,186 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,185 DEBUG [RS:1;asf911:34281] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,185 DEBUG [RS_CLOSE_REGION-asf911:34281-1] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:03,185 ERROR [main] helpers.MarkerIgnoringBase(159): killAll java.lang.Throwable at org.apache.hadoop.hbase.MiniHBaseCluster.killAll(MiniHBaseCluster.java:827) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:116) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:27:03,189 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'asf911.gq1.ygridcore.net,36180,1515828415912' ***** 2018-01-13 07:27:03,189 INFO [main] regionserver.HRegionServer(2172): STOPPED: Stopped by main 2018-01-13 07:27:03,185 INFO [RS:2;asf911:48364] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 07:27:03,181 DEBUG [RS-EventLoopGroup-41-8] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:50890 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:50890 2018-01-13 07:27:03,180 DEBUG [RS_CLOSE_META-asf911:60497-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 07:27:03,180 DEBUG [RS_CLOSE_REGION-asf911:34281-0] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347.: disabling compactions & flushes 2018-01-13 07:27:03,192 INFO [RS_CLOSE_META-asf911:60497-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 07:27:03,190 INFO [SplitLogWorker-asf911:48364] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 07:27:03,192 DEBUG [RS_CLOSE_META-asf911:60497-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 07:27:03,190 INFO [RS:2;asf911:48364] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 07:27:03,189 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,188 INFO [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:03,188 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1388): Waiting on 4 regions to close 2018-01-13 07:27:03,186 INFO [RS_CLOSE_REGION-asf911:60497-0] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:03,194 DEBUG [RS:1;asf911:34281] regionserver.HRegionServer(1392): Online Regions={7428bc7e2e038f99355fdf0180efc347=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347., 0a60e0b567378965cf87f0b02cf06c77=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77., 63847c70149ab9a684442e6301bf4c16=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.} 2018-01-13 07:27:03,194 DEBUG [RS_CLOSE_REGION-asf911:34281-2] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1515828420809.18f591905bbaa13514b7c6c437b3c57e. 2018-01-13 07:27:03,194 INFO [RS_CLOSE_REGION-asf911:34281-1] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:03,193 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 07:27:03,194 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 07:27:03,193 INFO [RS:2;asf911:48364] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-01-13 07:27:03,195 INFO [RS:2;asf911:48364] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-01-13 07:27:03,192 INFO [SplitLogWorker-asf911:48364] regionserver.SplitLogWorker(151): SplitLogWorker asf911.gq1.ygridcore.net,48364,1515828420610 exiting 2018-01-13 07:27:03,192 DEBUG [RS_CLOSE_REGION-asf911:34281-0] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:03,195 DEBUG [RS_CLOSE_REGION-asf911:34281-1] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1515828420809.0a60e0b567378965cf87f0b02cf06c77. 2018-01-13 07:27:03,195 DEBUG [RS_CLOSE_REGION-asf911:34281-2] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:03,195 DEBUG [RS_CLOSE_REGION-asf911:60497-0] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1515828420809.6cf6a059941c0ba105d47976a87fccb1. 2018-01-13 07:27:03,198 DEBUG [RS_CLOSE_REGION-asf911:48364-1] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:03,198 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1078): aborting server asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,200 DEBUG [RS:2;asf911:48364] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:27:03,198 DEBUG [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.: disabling compactions & flushes 2018-01-13 07:27:03,201 INFO [RS:2;asf911:48364] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x70186d8c to localhost:59094 2018-01-13 07:27:03,201 DEBUG [RS:2;asf911:48364] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,201 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1388): Waiting on 3 regions to close 2018-01-13 07:27:03,200 DEBUG [RS_CLOSE_REGION-asf911:48364-2] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:03,200 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,200 DEBUG [RS_CLOSE_REGION-asf911:48364-1] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d.: disabling compactions & flushes 2018-01-13 07:27:03,203 INFO [RS_CLOSE_REGION-asf911:34281-0] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:03,198 DEBUG [RS_CLOSE_REGION-asf911:48364-0] handler.CloseRegionHandler(92): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:03,203 DEBUG [RS_CLOSE_REGION-asf911:34281-0] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1515828420809.7428bc7e2e038f99355fdf0180efc347. 2018-01-13 07:27:03,203 DEBUG [RS_CLOSE_REGION-asf911:48364-1] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:03,203 DEBUG [RS_CLOSE_REGION-asf911:48364-2] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.: disabling compactions & flushes 2018-01-13 07:27:03,201 DEBUG [RS:2;asf911:48364] regionserver.HRegionServer(1392): Online Regions={b9b84001e287f4bc75307879516842fa=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa., 96f36daa2b54a675b901bda48069e68d=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d., 5d428ce1bfa5a483bf20d1264ccf3bd7=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.} 2018-01-13 07:27:03,201 DEBUG [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:03,206 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 07:27:03,205 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,207 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,205 DEBUG [RS_CLOSE_REGION-asf911:48364-2] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:03,205 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,205 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,205 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 07:27:03,204 DEBUG [RS_CLOSE_REGION-asf911:48364-0] regionserver.HRegion(1481): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa.: disabling compactions & flushes 2018-01-13 07:27:03,207 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,207 DEBUG [M:1;asf911:36180] util.FSTableDescriptors(787): Current tableInfoPath = hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 07:27:03,208 INFO [RS_CLOSE_REGION-asf911:48364-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:03,207 INFO [RS_CLOSE_REGION-asf911:34281-2] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:03,207 WARN [main] zookeeper.ZKUtil(618): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:27:03,208 ERROR [main] zookeeper.ZKWatcher(578): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:27:03,209 ERROR [main] master.ActiveMasterManager(280): master:50890-0x160ee6894500078, quorum=localhost:59094, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:340) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:610) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:150) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:269) at org.apache.hadoop.hbase.master.HMaster.stop(HMaster.java:2689) at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2682) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:398) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:623) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:1113) at org.apache.hadoop.hbase.TestZooKeeper.after(TestZooKeeper.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) 2018-01-13 07:27:03,206 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,208 DEBUG [RS_CLOSE_REGION-asf911:34281-2] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1515828420809.63847c70149ab9a684442e6301bf4c16. 2018-01-13 07:27:03,209 INFO [RS_CLOSE_REGION-asf911:48364-1] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:03,208 DEBUG [RS_CLOSE_REGION-asf911:48364-2] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1515828420809.5d428ce1bfa5a483bf20d1264ccf3bd7. 2018-01-13 07:27:03,207 DEBUG [RS_CLOSE_REGION-asf911:48364-0] regionserver.HRegion(1520): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:03,210 DEBUG [RS_CLOSE_REGION-asf911:48364-1] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1515828420809.96f36daa2b54a675b901bda48069e68d. 2018-01-13 07:27:03,210 INFO [StoreCloserThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa.-1] regionserver.HStore(930): Closed info 2018-01-13 07:27:03,210 INFO [RS_CLOSE_REGION-asf911:48364-0] regionserver.HRegion(1633): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:03,210 DEBUG [RS_CLOSE_REGION-asf911:48364-0] handler.CloseRegionHandler(125): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1515828420809.b9b84001e287f4bc75307879516842fa. 2018-01-13 07:27:03,211 DEBUG [M:1;asf911:36180] util.FSTableDescriptors(791): TableInfo already exists.. Skipping creation 2018-01-13 07:27:03,212 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,212 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,212 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,212 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 07:27:03,212 DEBUG [main] zookeeper.ZKUtil(614): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:27:03,213 WARN [main] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:27:03,226 INFO [M:1;asf911:36180] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:27:03,227 INFO [M:1;asf911:36180] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 07:27:03,227 DEBUG [M:1;asf911:36180] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 07:27:03,229 INFO [asf911.gq1.ygridcore.net,36180,1515828415912_splitLogManager__Chore_1] hbase.ScheduledChore(180): Chore: SplitLogManager Timeout Monitor was stopped 2018-01-13 07:27:03,250 INFO [M:1;asf911:36180] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2a74d89a to localhost:59094, session timeout 1000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 07:27:03,258 DEBUG [M:1;asf911:36180] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17ab2fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 07:27:03,280 INFO [M:1;asf911:36180] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 07:27:03,280 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 07:27:03,281 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 07:27:03,282 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 07:27:03,282 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(357): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 07:27:03,282 INFO [M:1;asf911:36180] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 07:27:03,284 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,284 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,285 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(355): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,287 INFO [M:1;asf911:36180] master.HMaster(746): Server active/primary master=asf911.gq1.ygridcore.net,36180,1515828415912, sessionid=0x160ee6894500079, setting cluster-up flag (Was=true) 2018-01-13 07:27:03,296 DEBUG [M:1;asf911:36180] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 07:27:03,297 INFO [M:1;asf911:36180] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 07:27:03,298 DEBUG [M:1;asf911:36180] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:27:03,321 DEBUG [M:1;asf911:36180] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 07:27:03,321 INFO [M:1;asf911:36180] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 07:27:03,323 DEBUG [M:1;asf911:36180] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:27:03,324 INFO [M:1;asf911:36180] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-asf911:36180, corePoolSize=5, maxPoolSize=5 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-asf911:36180, corePoolSize=5, maxPoolSize=5 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-asf911:36180, corePoolSize=5, maxPoolSize=5 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf911:36180, corePoolSize=5, maxPoolSize=5 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-asf911:36180, corePoolSize=10, maxPoolSize=10 2018-01-13 07:27:03,325 DEBUG [M:1;asf911:36180] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-asf911:36180, corePoolSize=1, maxPoolSize=1 2018-01-13 07:27:03,327 INFO [M:1;asf911:36180] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 07:27:03,327 INFO [M:1;asf911:36180] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 07:27:03,327 INFO [M:1;asf911:36180] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log 2018-01-13 07:27:03,329 INFO [M:1;asf911:36180] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-01-13 07:27:03,329 DEBUG [M:1;asf911:36180] wal.WALProcedureStore(1286): Opening Pv2 FileStatus{path=hdfs://localhost:45316/user/jenkins/test-data/2192094b-6964-4130-b9eb-90194ec101f9/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=38685; replication=2; blocksize=134217728; modification_time=1515828423154; access_time=1515828416395; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-01-13 07:27:03,336 DEBUG [M:1;asf911:36180] wal.WALProcedureStore(1071): Roll new state log: 2 2018-01-13 07:27:03,337 INFO [M:1;asf911:36180] wal.WALProcedureStore(390): Lease acquired for flushLogId: 2 2018-01-13 07:27:03,337 INFO [M:1;asf911:36180] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 10msec 2018-01-13 07:27:03,343 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor$1(317): Load max pid=18 2018-01-13 07:27:03,344 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor(356): Completed pid=7, state=SUCCESS; CreateTableProcedure table=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2018-01-13 07:27:03,344 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor(356): Completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-13 07:27:03,345 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor(356): Completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-13 07:27:03,345 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor(356): Completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-13 07:27:03,345 DEBUG [M:1;asf911:36180] procedure2.ProcedureExecutor(356): Completed pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 07:27:03,346 INFO [M:1;asf911:36180] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 9msec 2018-01-13 07:27:03,347 INFO [M:1;asf911:36180] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515828453347 2018-01-13 07:27:03,347 INFO [M:1;asf911:36180] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 07:27:03,347 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 07:27:03,348 INFO [M:1;asf911:36180] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:27:03,348 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 07:27:03,348 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 07:27:03,348 INFO [M:1;asf911:36180] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 07:27:03,351 INFO [asf911.gq1.ygridcore.net,36180,1515828415912_Chore_1] master.ReplicationLogCleaner(121): Stopping replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase 2018-01-13 07:27:03,353 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 07:27:03,353 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 07:27:03,353 DEBUG [M:1;asf911:36180] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 07:27:03,354 DEBUG [M:1;asf911:36180] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:1;asf911:36180-HFileCleaner.large.0-1515828423354,5,main] 2018-01-13 07:27:03,354 DEBUG [M:1;asf911:36180] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:1;asf911:36180-HFileCleaner.small.0-1515828423354,5,main] 2018-01-13 07:27:03,354 INFO [M:1;asf911:36180] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515828416066, completionTime=-1 2018-01-13 07:27:03,354 INFO [M:1;asf911:36180] master.ServerManager(891): Finished wait on RegionServer count=0; waited=0ms, expected min=2 server(s), max=2 server(s), master is stopped. 2018-01-13 07:27:03,354 INFO [M:1;asf911:36180] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,354 INFO [M:1;asf911:36180] master.HMaster(1048): Registered server found up in zk but who has not yet reported in: asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,355 INFO [M:1;asf911:36180] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,355 INFO [M:1;asf911:36180] master.HMaster(1048): Registered server found up in zk but who has not yet reported in: asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,355 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,60497,1515828415979; all regions closed. 2018-01-13 07:27:03,355 INFO [M:1;asf911:36180] master.ServerManager(432): Registering server=asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,355 INFO [M:1;asf911:36180] master.HMaster(1048): Registered server found up in zk but who has not yet reported in: asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,360 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741939_1117{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:03,360 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741939_1117{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:03,363 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741937_1115{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW]]} size 0 2018-01-13 07:27:03,364 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741937_1115 size 2210 2018-01-13 07:27:03,364 DEBUG [RS:0;asf911:60497] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,365 INFO [RS:0;asf911:60497] regionserver.Leases(147): RS:0;asf911:60497 closing leases 2018-01-13 07:27:03,365 INFO [RS:0;asf911:60497] regionserver.Leases(150): RS:0;asf911:60497 closed leases 2018-01-13 07:27:03,365 INFO [RS:0;asf911:60497] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,60497,1515828415979 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,60497,1515828415979 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:27:03,365 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:27:03,366 INFO [RS:0;asf911:60497] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:27:03,379 DEBUG [M:1;asf911:36180-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:27:03,395 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,34281,1515828416028; all regions closed. 2018-01-13 07:27:03,403 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741938_1116{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:03,403 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741938_1116{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:03,404 DEBUG [RS:1;asf911:34281] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,404 INFO [RS:1;asf911:34281] regionserver.Leases(147): RS:1;asf911:34281 closing leases 2018-01-13 07:27:03,404 INFO [RS:1;asf911:34281] regionserver.Leases(150): RS:1;asf911:34281 closed leases 2018-01-13 07:27:03,405 INFO [RS:1;asf911:34281] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,34281,1515828416028 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,34281,1515828416028 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:27:03,405 INFO [RS:1;asf911:34281] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:27:03,405 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,48364,1515828420610; all regions closed. 2018-01-13 07:27:03,405 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:27:03,405 INFO [RS:1;asf911:34281] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:27:03,405 INFO [RS:1;asf911:34281] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:27:03,406 INFO [RS:1;asf911:34281] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:27:03,409 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,409 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,409 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:03,409 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,410 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,410 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:03,415 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,60497,1515828415979] 2018-01-13 07:27:03,410 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:03,417 INFO [main-EventThread] master.ServerManager(573): Master doesn't enable ServerShutdownHandler during initialization, delay expiring server asf911.gq1.ygridcore.net,60497,1515828415979 2018-01-13 07:27:03,417 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(532): Error while calling watcher java.lang.NullPointerException at org.apache.hadoop.hbase.master.assignment.AssignmentManager.handleMetaRITOnCrashedServer(AssignmentManager.java:1912) at org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:578) at org.apache.hadoop.hbase.master.RegionServerTracker.nodeDeleted(RegionServerTracker.java:131) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,417 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:03,418 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741954_1132{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:03,418 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741954_1132{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-c601089d-c719-49d5-8041-8d8fdc76b7cc:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:03,419 DEBUG [RS:2;asf911:48364] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:03,419 INFO [RS:2;asf911:48364] regionserver.Leases(147): RS:2;asf911:48364 closing leases 2018-01-13 07:27:03,419 INFO [RS:2;asf911:48364] regionserver.Leases(150): RS:2;asf911:48364 closed leases 2018-01-13 07:27:03,419 INFO [RS:2;asf911:48364] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,48364,1515828420610 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,48364,1515828420610 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 07:27:03,420 INFO [RS:2;asf911:48364] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 07:27:03,420 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 07:27:03,420 INFO [RS:2;asf911:48364] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 07:27:03,420 INFO [RS:2;asf911:48364] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 07:27:03,420 INFO [RS:2;asf911:48364] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:27:03,423 DEBUG [M:1;asf911:36180-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ee6894500087 connected 2018-01-13 07:27:03,424 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:60497-0x160ee689450007a, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,424 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,424 DEBUG [OldWALsCleaner-0] cleaner.LogCleaner(176): Exiting cleaner. 2018-01-13 07:27:03,424 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,424 INFO [asf911.gq1.ygridcore.net,36180,1515828415912_Chore_1] hbase.ScheduledChore(180): Chore: LogsCleaner was stopped 2018-01-13 07:27:03,424 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,34281,1515828416028] 2018-01-13 07:27:03,424 DEBUG [OldWALsCleaner-1] cleaner.LogCleaner(176): Exiting cleaner. 2018-01-13 07:27:03,424 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,424 INFO [main-EventThread] master.ServerManager(573): Master doesn't enable ServerShutdownHandler during initialization, delay expiring server asf911.gq1.ygridcore.net,34281,1515828416028 2018-01-13 07:27:03,424 DEBUG [asf911.gq1.ygridcore.net,36180,1515828415912_Chore_1] cleaner.HFileCleaner(282): Stopping file delete threads 2018-01-13 07:27:03,426 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(532): Error while calling watcher java.lang.NullPointerException at org.apache.hadoop.hbase.master.assignment.AssignmentManager.handleMetaRITOnCrashedServer(AssignmentManager.java:1912) at org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:578) at org.apache.hadoop.hbase.master.RegionServerTracker.nodeDeleted(RegionServerTracker.java:131) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,426 INFO [asf911.gq1.ygridcore.net,36180,1515828415912_Chore_1] hbase.ScheduledChore(180): Chore: HFileCleaner was stopped 2018-01-13 07:27:03,426 INFO [asf911.gq1.ygridcore.net,36180,1515828415912_Chore_1] hbase.ScheduledChore(180): Chore: ReplicationMetaCleaner was stopped 2018-01-13 07:27:03,426 DEBUG [M:1;asf911:36180-HFileCleaner.small.0-1515828423354] cleaner.HFileCleaner(234): Interrupted while trying to take a task from queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) 2018-01-13 07:27:03,426 DEBUG [M:1;asf911:36180-HFileCleaner.small.0-1515828423354] cleaner.HFileCleaner(253): Exit thread: Thread[M:1;asf911:36180-HFileCleaner.small.0-1515828423354,5,main] 2018-01-13 07:27:03,426 DEBUG [M:1;asf911:36180-HFileCleaner.large.0-1515828423354] cleaner.HFileCleaner(234): Interrupted while trying to take a task from queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) 2018-01-13 07:27:03,428 DEBUG [M:1;asf911:36180-HFileCleaner.large.0-1515828423354] cleaner.HFileCleaner(253): Exit thread: Thread[M:1;asf911:36180-HFileCleaner.large.0-1515828423354,5,main] 2018-01-13 07:27:03,432 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,60497,1515828415979; zookeeper connection closed. 2018-01-13 07:27:03,432 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(460): regionserver:48364-0x160ee6894500083, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,432 INFO [RS:0;asf911:60497] regionserver.HRegionServer(1157): RS:0;asf911:60497 exiting 2018-01-13 07:27:03,432 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:34281-0x160ee689450007b, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,432 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,432 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,48364,1515828420610] 2018-01-13 07:27:03,432 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7596e7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7596e7 2018-01-13 07:27:03,432 INFO [main-EventThread] master.ServerManager(573): Master doesn't enable ServerShutdownHandler during initialization, delay expiring server asf911.gq1.ygridcore.net,48364,1515828420610 2018-01-13 07:27:03,433 ERROR [main-EventThread] zookeeper.ClientCnxn$EventThread(532): Error while calling watcher java.lang.NullPointerException at org.apache.hadoop.hbase.master.assignment.AssignmentManager.handleMetaRITOnCrashedServer(AssignmentManager.java:1912) at org.apache.hadoop.hbase.master.ServerManager.expireServer(ServerManager.java:578) at org.apache.hadoop.hbase.master.RegionServerTracker.nodeDeleted(RegionServerTracker.java:131) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 07:27:03,440 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,34281,1515828416028; zookeeper connection closed. 2018-01-13 07:27:03,440 INFO [RS:1;asf911:34281] regionserver.HRegionServer(1157): RS:1;asf911:34281 exiting 2018-01-13 07:27:03,441 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@69ba57a1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@69ba57a1 2018-01-13 07:27:03,449 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,48364,1515828420610; zookeeper connection closed. 2018-01-13 07:27:03,449 INFO [RS:2;asf911:48364] regionserver.HRegionServer(1157): RS:2;asf911:48364 exiting 2018-01-13 07:27:03,457 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1ae34349] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1ae34349 2018-01-13 07:27:03,457 INFO [main] util.JVMClusterUtil(322): Shutdown of 2 master(s) and 3 regionserver(s) complete 2018-01-13 07:27:03,722 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 07:27:03,749 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:27:03,749 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:27:03,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:27:03,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(147): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closing leases 2018-01-13 07:27:03,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:27:03,907 INFO [regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker] regionserver.Leases(150): regionserver/asf911.gq1.ygridcore.net/67.195.81.155:0.leaseChecker closed leases 2018-01-13 07:27:04,383 INFO [M:1;asf911:36180] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 07:27:04,384 INFO [M:1;asf911:36180] regionserver.HRegionServer(1081): stopping server asf911.gq1.ygridcore.net,36180,1515828415912 2018-01-13 07:27:04,384 DEBUG [M:1;asf911:36180] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 07:27:04,384 INFO [M:1;asf911:36180] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x2a74d89a to localhost:59094 2018-01-13 07:27:04,384 DEBUG [M:1;asf911:36180] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:04,384 INFO [M:1;asf911:36180] regionserver.HRegionServer(1109): stopping server asf911.gq1.ygridcore.net,36180,1515828415912; all regions closed. 2018-01-13 07:27:04,384 INFO [M:1;asf911:36180] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,36180,1515828415912 had [] on shutdown 2018-01-13 07:27:04,384 DEBUG [M:1;asf911:36180] master.HMaster(1187): Stopping service threads 2018-01-13 07:27:04,399 DEBUG [M:1;asf911:36180] zookeeper.ZKUtil(614): master:36180-0x160ee6894500079, quorum=localhost:59094, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-01-13 07:27:04,399 WARN [M:1;asf911:36180] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-01-13 07:27:04,399 INFO [M:1;asf911:36180] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-13 07:27:04,399 INFO [M:1;asf911:36180] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-13 07:27:04,399 INFO [M:1;asf911:36180] procedure2.ProcedureExecutor(572): Stopping 2018-01-13 07:27:04,402 INFO [M:1;asf911:36180] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-13 07:27:04,402 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,402 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,410 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,411 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,411 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,410 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,410 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,412 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,411 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,411 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-13 07:27:04,421 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49352 is added to blk_1073741955_1133{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c2f91fd0-9011-4287-bd13-96644a3511e6:NORMAL:127.0.0.1:59189|RBW], ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW]]} size 0 2018-01-13 07:27:04,421 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59189 is added to blk_1073741955_1133{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9a2f1cda-d6f3-480f-a4bf-8358260a3189:NORMAL:127.0.0.1:49352|RBW], ReplicaUC[[DISK]DS-5ba0c480-698e-4fa2-be8d-55b114c682ac:NORMAL:127.0.0.1:59189|FINALIZED]]} size 0 2018-01-13 07:27:04,422 INFO [M:1;asf911:36180] hbase.ChoreService(327): Chore service for: asf911.gq1.ygridcore.net,36180,1515828415912_splitLogManager_ had [] on shutdown 2018-01-13 07:27:04,422 INFO [M:1;asf911:36180] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-13 07:27:04,432 DEBUG [M:1;asf911:36180] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,36180,1515828415912 already deleted, retry=false 2018-01-13 07:27:04,440 INFO [M:1;asf911:36180] regionserver.HRegionServer(1155): stopping server asf911.gq1.ygridcore.net,36180,1515828415912; zookeeper connection closed. 2018-01-13 07:27:04,440 INFO [M:1;asf911:36180] regionserver.HRegionServer(1157): M:1;asf911:36180 exiting 2018-01-13 07:27:04,449 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741936_1114 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741955_1133 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741938_1116 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741954_1132 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741937_1115 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741939_1117 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,452 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741942_1120 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741945_1123 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741944_1122 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741947_1125 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741946_1124 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741943_1121 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741950_1128 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,453 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741948_1126 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,454 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741953_1131 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,454 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741952_1130 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,454 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741951_1129 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,454 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741949_1127 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,454 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741935_1113 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,455 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741934_1112 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,455 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741940_1118 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,455 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741941_1119 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,455 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741933_1111 127.0.0.1:59189 127.0.0.1:49352 2018-01-13 07:27:04,455 INFO [IPC Server handler 8 on 45316] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741932_1110 127.0.0.1:49352 127.0.0.1:59189 2018-01-13 07:27:04,456 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=testing utility connecting to ZooKeeper ensemble=localhost:59094 2018-01-13 07:27:04,465 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility0x0, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 07:27:04,468 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): testing utility-0x160ee6894500088 connected 2018-01-13 07:27:04,469 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2018-01-13 07:27:04,469 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/rs 2018-01-13 07:27:04,470 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/splitWAL 2018-01-13 07:27:04,471 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters 2018-01-13 07:27:04,472 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table-lock 2018-01-13 07:27:04,472 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc 2018-01-13 07:27:04,473 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/master-maintenance 2018-01-13 07:27:04,473 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot 2018-01-13 07:27:04,474 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/switch 2018-01-13 07:27:04,475 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-01-13 07:27:04,475 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/draining 2018-01-13 07:27:04,476 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2018-01-13 07:27:04,476 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/hbaseid 2018-01-13 07:27:04,477 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/table 2018-01-13 07:27:04,480 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/abort 2018-01-13 07:27:04,480 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired 2018-01-13 07:27:04,481 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached 2018-01-13 07:27:04,482 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/abort 2018-01-13 07:27:04,483 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/acquired 2018-01-13 07:27:04,484 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/online-snapshot/reached 2018-01-13 07:27:04,486 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/default 2018-01-13 07:27:04,486 DEBUG [main] zookeeper.ZKUtil(355): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace/hbase 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/reached 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/acquired 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot/abort 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached 2018-01-13 07:27:04,499 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/hbaseid 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/namespace 2018-01-13 07:27:04,500 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/draining 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/switch 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/online-snapshot 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master-maintenance 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc 2018-01-13 07:27:04,509 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/table-lock 2018-01-13 07:27:04,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters 2018-01-13 07:27:04,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL 2018-01-13 07:27:04,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs 2018-01-13 07:27:04,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 07:27:04,510 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): testing utility-0x160ee6894500088, quorum=localhost:59094, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase 2018-01-13 07:27:04,553 INFO [main] hbase.ResourceChecker(172): after: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=697 (was 641) Potentially hanging thread: RS-EventLoopGroup-37-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-38-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-18 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-39-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-39-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1896267467_1 at /127.0.0.1:55449 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:50890-SendThread(localhost:59094) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: RS-EventLoopGroup-37-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.17 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-41-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-20 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.17@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-39-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:50890-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RS-EventLoopGroup-40-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-19 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-39-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-37-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: M:0;asf911:50890-HFileCleaner.small.0-1515828416406 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:216) Potentially hanging thread: M:0;asf911:50890-HFileCleaner.large.0-1515828416405 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:232) org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:201) Potentially hanging thread: RS-EventLoopGroup-40-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-37-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-21 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.19@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: globalEventExecutor-2-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor.takeTask(GlobalEventExecutor.java:90) org.apache.hbase.thirdparty.io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:230) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-37-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.19 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS-EventLoopGroup-41-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-40-17 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.18@localhost:45316 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-39-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-41-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1896267467_1 at /127.0.0.1:41264 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (542682055) connection to localhost/127.0.0.1:45316 from jenkins.hfs.18 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) - Thread LEAK? -, OpenFileDescriptor=4694 (was 4187) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=312 (was 322), ProcessCount=270 (was 271), AvailableMemoryMB=9896 (was 10132) 2018-01-13 07:27:04,554 WARN [main] hbase.ResourceChecker(135): Thread=697 is superior to 500 2018-01-13 07:27:04,554 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=4694 is superior to 1024 2018-01-13 07:27:04,554 INFO [main] hbase.HBaseTestingUtility(1085): Shutting down minicluster 2018-01-13 07:27:04,554 INFO [main] client.ConnectionImplementation(1763): Closing master protocol: MasterService 2018-01-13 07:27:04,555 INFO [main] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x1a3e8e24 to localhost:59094 2018-01-13 07:27:04,555 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 07:27:04,574 WARN [main] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-01-13 07:27:04,595 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-01-13 07:27:04,702 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:45316] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-01-13 07:27:04,702 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:45316] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1237337564-67.195.81.155-1515828314954 (Datanode Uuid f9f33522-cf14-4d6d-9f8d-50a8a8519fd9) service to localhost/127.0.0.1:45316 2018-01-13 07:27:04,714 WARN [main] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-01-13 07:27:04,733 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-01-13 07:27:04,838 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@749c877b] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:59189 to delete [blk_1073741952_1130, blk_1073741953_1131, blk_1073741954_1132, blk_1073741955_1133, blk_1073741932_1110, blk_1073741933_1111, blk_1073741934_1112, blk_1073741935_1113, blk_1073741936_1114, blk_1073741937_1115, blk_1073741938_1116, blk_1073741939_1117, blk_1073741940_1118, blk_1073741941_1119, blk_1073741942_1120, blk_1073741943_1121, blk_1073741944_1122, blk_1073741945_1123, blk_1073741946_1124, blk_1073741947_1125, blk_1073741948_1126, blk_1073741949_1127, blk_1073741950_1128, blk_1073741951_1129] 2018-01-13 07:27:04,838 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:45316] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-01-13 07:27:04,839 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/2c7648e7-8dff-4db9-b5f8-4f1b7a8536e0/cluster_1aa16553-f2a4-4a5b-bdd5-d72e68250458/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:45316] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1237337564-67.195.81.155-1515828314954 (Datanode Uuid e361b17c-883b-4f18-a1bd-92fb43f01ad1) service to localhost/127.0.0.1:45316 2018-01-13 07:27:04,876 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-01-13 07:27:04,989 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-13 07:27:04,991 INFO [main] zookeeper.MiniZooKeeperCluster(324): Shutdown MiniZK cluster with all ZK servers 2018-01-13 07:27:05,015 INFO [main] hbase.HBaseTestingUtility(1096): Minicluster is down 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,027 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,028 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@e4487af 2018-01-13 07:27:05,030 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-01-13 07:27:05,031 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.