2018-05-25 02:51:02,119 DEBUG [main] hbase.HBaseTestingUtility(344): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63 2018-05-25 02:51:02,129 INFO [Time-limited test] master.TestAssignmentManagerMetrics(73): Starting cluster 2018-05-25 02:51:02,130 INFO [Time-limited test] hbase.HBaseTestingUtility(954): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2018-05-25 02:51:02,131 INFO [Time-limited test] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf, deleteOnExit=true 2018-05-25 02:51:02,131 INFO [Time-limited test] hbase.HBaseTestingUtility(969): STARTING DFS 2018-05-25 02:51:02,307 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cache_data in system properties and HBase conf 2018-05-25 02:51:02,308 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/hadoop_tmp in system properties and HBase conf 2018-05-25 02:51:02,309 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/hadoop_logs in system properties and HBase conf 2018-05-25 02:51:02,309 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/mapred_local in system properties and HBase conf 2018-05-25 02:51:02,310 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/mapred_temp in system properties and HBase conf 2018-05-25 02:51:02,310 INFO [Time-limited test] hbase.HBaseTestingUtility(737): read short circuit is OFF 2018-05-25 02:51:02,436 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-05-25 02:51:02,897 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-05-25 02:51:05,102 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-05-25 02:51:05,359 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-05-25 02:51:05,451 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-05-25 02:51:05,495 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_56943_hdfs____.xi7wnz/webapp 2018-05-25 02:51:05,713 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56943 2018-05-25 02:51:06,871 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-05-25 02:51:06,880 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_39347_datanode____.6tblmx/webapp 2018-05-25 02:51:07,022 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39347 2018-05-25 02:51:09,173 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:43302] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-05-25 02:51:09,266 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x122d41d68ec4fe: from storage DS-6a595950-4d09-4fed-930f-9a9193bb9c36 node DatanodeRegistration(127.0.0.1:42355, datanodeUuid=1621ce37-98d3-4d8e-87e2-8c5d539f68c6, infoPort=48322, infoSecurePort=0, ipcPort=56929, storageInfo=lv=-56;cid=testClusterID;nsid=235824926;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs 2018-05-25 02:51:09,266 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x122d41d68ec4fe: from storage DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786 node DatanodeRegistration(127.0.0.1:42355, datanodeUuid=1621ce37-98d3-4d8e-87e2-8c5d539f68c6, infoPort=48322, infoSecurePort=0, ipcPort=56929, storageInfo=lv=-56;cid=testClusterID;nsid=235824926;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-05-25 02:51:09,335 DEBUG [Time-limited test] hbase.HBaseTestingUtility(672): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63 2018-05-25 02:51:09,401 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-05-25 02:51:09,418 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=53584 2018-05-25 02:51:09,434 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:09,439 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:09,902 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 7 2018-05-25 02:51:10,331 INFO [Time-limited test] util.FSUtils(515): Created version file at hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0 with version=8 2018-05-25 02:51:10,331 INFO [Time-limited test] hbase.HBaseTestingUtility(1213): Setting hbase.fs.tmp.dir to hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/hbase-staging 2018-05-25 02:51:10,618 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-05-25 02:51:11,104 INFO [Time-limited test] client.ConnectionUtils(122): master/asf911:0 server-side Connection retries=3 2018-05-25 02:51:11,135 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-05-25 02:51:11,137 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-05-25 02:51:11,137 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-05-25 02:51:11,297 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-05-25 02:51:11,506 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-05-25 02:51:11,610 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:46610 2018-05-25 02:51:11,628 INFO [Time-limited test] hfile.CacheConfig(553): Allocating onheap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-05-25 02:51:11,637 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:11,639 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:11,645 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-05-25 02:51:11,649 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:11,653 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:11,721 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:46610 connecting to ZooKeeper ensemble=localhost:53584 2018-05-25 02:51:12,295 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:466100x0, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-05-25 02:51:12,297 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(541): master:46610-0x163953502c90000 connected 2018-05-25 02:51:12,576 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-05-25 02:51:12,579 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-05-25 02:51:12,594 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46610 2018-05-25 02:51:12,596 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=46610 2018-05-25 02:51:12,597 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46610 2018-05-25 02:51:12,607 INFO [Time-limited test] master.HMaster(491): hbase.rootdir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0, hbase.cluster.distributed=false 2018-05-25 02:51:12,719 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=3 2018-05-25 02:51:12,720 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-05-25 02:51:12,720 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-05-25 02:51:12,720 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-05-25 02:51:12,725 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-05-25 02:51:12,725 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-05-25 02:51:12,729 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:36079 2018-05-25 02:51:12,730 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:12,731 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:12,733 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:12,737 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:12,740 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:36079 connecting to ZooKeeper ensemble=localhost:53584 2018-05-25 02:51:12,757 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:360790x0, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-05-25 02:51:12,758 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(541): regionserver:36079-0x163953502c90001 connected 2018-05-25 02:51:12,760 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-05-25 02:51:12,763 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-05-25 02:51:12,764 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36079 2018-05-25 02:51:12,765 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=36079 2018-05-25 02:51:12,766 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36079 2018-05-25 02:51:12,771 INFO [Thread-81] master.HMaster(2039): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:12,843 DEBUG [Thread-81] zookeeper.ZKUtil(355): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:12,900 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-05-25 02:51:12,900 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-05-25 02:51:12,904 DEBUG [Thread-81] zookeeper.ZKUtil(355): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-05-25 02:51:12,908 INFO [Thread-81] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,46610,1527216670401 from backup master directory 2018-05-25 02:51:12,916 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-05-25 02:51:12,957 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:12,958 WARN [Thread-81] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-05-25 02:51:12,958 INFO [Thread-81] master.ActiveMasterManager(181): Registered as active master=asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:12,963 INFO [Thread-81] regionserver.ChunkCreator(498): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-05-25 02:51:12,965 INFO [Thread-81] regionserver.ChunkCreator(498): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-05-25 02:51:13,205 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|FINALIZED]]} size 0 2018-05-25 02:51:13,225 DEBUG [Thread-81] util.FSUtils(667): Created cluster ID file at hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/hbase.id with ID: ece95e6b-d74a-4d96-9d37-0fe98752b37e 2018-05-25 02:51:13,271 INFO [Thread-81] master.MasterFileSystem(394): BOOTSTRAP: creating hbase:meta region 2018-05-25 02:51:13,278 INFO [Thread-81] regionserver.HRegion(6911): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0 Table name == hbase:meta 2018-05-25 02:51:13,317 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|FINALIZED]]} size 0 2018-05-25 02:51:13,357 DEBUG [Thread-81] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:13,422 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/info 2018-05-25 02:51:13,445 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:13,458 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:13,479 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:13,485 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/rep_barrier 2018-05-25 02:51:13,487 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:13,488 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:13,490 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:13,496 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/table 2018-05-25 02:51:13,497 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:13,498 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:13,500 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:13,513 DEBUG [Thread-81] regionserver.HRegion(4469): Found 0 recovered edits file(s) under hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740 2018-05-25 02:51:13,534 DEBUG [Thread-81] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-05-25 02:51:13,564 DEBUG [Thread-81] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-05-25 02:51:13,564 INFO [Thread-81] regionserver.HRegion(975): Opened 1588230740; next sequenceid=2 2018-05-25 02:51:13,565 DEBUG [Thread-81] regionserver.HRegion(1514): Closing 1588230740, disabling compactions & flushes 2018-05-25 02:51:13,565 DEBUG [Thread-81] regionserver.HRegion(1554): Updates disabled for region hbase:meta,,1.1588230740 2018-05-25 02:51:13,566 INFO [Thread-81] regionserver.HRegion(1668): Closed hbase:meta,,1.1588230740 2018-05-25 02:51:13,636 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 0 2018-05-25 02:51:13,651 DEBUG [Thread-81] util.FSTableDescriptors(683): Wrote into hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-05-25 02:51:13,748 INFO [Thread-81] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-05-25 02:51:13,766 INFO [Thread-81] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-05-25 02:51:13,826 INFO [Thread-81] zookeeper.ReadOnlyZKClient(139): Connect 0x4c970a31 to localhost:53584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-05-25 02:51:13,879 DEBUG [Thread-81] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d141b8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-05-25 02:51:13,921 INFO [Thread-81] balancer.BaseLoadBalancer(1039): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-05-25 02:51:13,929 INFO [Thread-81] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-05-25 02:51:13,954 DEBUG [Thread-81] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-05-25 02:51:13,956 DEBUG [Thread-81] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-05-25 02:51:13,975 DEBUG [Thread-81] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-05-25 02:51:13,976 DEBUG [Thread-81] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-05-25 02:51:14,056 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-05-25 02:51:14,056 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-05-25 02:51:14,057 INFO [Thread-81] master.HMaster(782): Active/primary master=asf911.gq1.ygridcore.net,46610,1527216670401, sessionid=0x163953502c90000, setting cluster-up flag (Was=false) 2018-05-25 02:51:14,173 DEBUG [Thread-81] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-05-25 02:51:14,176 DEBUG [Thread-81] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:14,280 DEBUG [Thread-81] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-05-25 02:51:14,283 DEBUG [Thread-81] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:14,345 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-05-25 02:51:14,345 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-05-25 02:51:14,345 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-05-25 02:51:14,345 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-05-25 02:51:14,346 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-master/asf911:0, corePoolSize=10, maxPoolSize=10 2018-05-25 02:51:14,346 DEBUG [Thread-81] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-master/asf911:0, corePoolSize=1, maxPoolSize=1 2018-05-25 02:51:14,374 INFO [RS:0;asf911:36079] regionserver.HRegionServer(874): ClusterId : ece95e6b-d74a-4d96-9d37-0fe98752b37e 2018-05-25 02:51:14,380 INFO [Thread-81] procedure2.ProcedureExecutor(533): Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160 2018-05-25 02:51:14,380 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-05-25 02:51:14,394 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-05-25 02:51:14,394 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-05-25 02:51:14,394 WARN [Thread-81] util.CommonFSUtils$StreamCapabilities(830): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-05-25 02:51:14,396 DEBUG [Thread-81] util.CommonFSUtils$StreamCapabilities(837): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:826) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:864) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1042) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:382) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:550) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1278) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1184) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:881) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2059) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:14,402 DEBUG [Thread-81] wal.WALProcedureStore(1078): Roll new state log: 1 2018-05-25 02:51:14,404 INFO [Thread-81] procedure2.ProcedureExecutor(552): Recovered WALProcedureStore lease in 21msec 2018-05-25 02:51:14,406 INFO [Thread-81] procedure2.ProcedureExecutor(566): Loaded WALProcedureStore in 1msec 2018-05-25 02:51:14,407 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-05-25 02:51:14,410 INFO [RS:0;asf911:36079] zookeeper.ReadOnlyZKClient(139): Connect 0x5bdfe4e5 to localhost:53584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-05-25 02:51:14,411 INFO [Thread-81] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1527216704410 2018-05-25 02:51:14,412 INFO [Thread-81] procedure2.RemoteProcedureDispatcher(97): Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-05-25 02:51:14,417 INFO [Thread-81] cleaner.CleanerChore$DirScanPool(87): Cleaner pool size is 4 2018-05-25 02:51:14,419 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-05-25 02:51:14,420 INFO [Thread-81] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:53584 2018-05-25 02:51:14,422 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-05-25 02:51:14,423 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-05-25 02:51:14,423 INFO [Thread-81] cleaner.LogCleaner(122): Creating OldWALs cleaners with size=2 2018-05-25 02:51:14,427 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-05-25 02:51:14,430 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-05-25 02:51:14,430 DEBUG [Thread-81] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-05-25 02:51:14,432 DEBUG [Thread-81] cleaner.HFileCleaner(207): Starting for large file=Thread[Thread-81-HFileCleaner.large.0-1527216674432,5,FailOnTimeoutGroup] 2018-05-25 02:51:14,432 DEBUG [Thread-81] cleaner.HFileCleaner(222): Starting for small files=Thread[Thread-81-HFileCleaner.small.0-1527216674432,5,FailOnTimeoutGroup] 2018-05-25 02:51:14,434 INFO [Thread-81] master.HMaster(891): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1527216672869, completionTime=-1 2018-05-25 02:51:14,434 INFO [Thread-81] master.ServerManager(867): Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=1 server(s), timeout=4500ms, lastChange=0ms 2018-05-25 02:51:14,438 DEBUG [Thread-81-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner0x0, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-05-25 02:51:14,439 DEBUG [Thread-81-EventThread] zookeeper.ZKWatcher(541): replicationLogCleaner-0x163953502c90004 connected 2018-05-25 02:51:14,440 DEBUG [RS:0;asf911:36079] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@551801ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-05-25 02:51:14,441 DEBUG [RS:0;asf911:36079] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51278f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-05-25 02:51:14,444 DEBUG [RS:0;asf911:36079] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:36079 2018-05-25 02:51:14,450 INFO [RS:0;asf911:36079] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-05-25 02:51:14,450 INFO [RS:0;asf911:36079] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-05-25 02:51:14,454 INFO [RS:0;asf911:36079] regionserver.HRegionServer(2595): reportForDuty to master=asf911.gq1.ygridcore.net,46610,1527216670401 with port=36079, startcode=1527216672716 2018-05-25 02:51:14,565 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.0 (auth:SIMPLE) 2018-05-25 02:51:14,574 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 42563 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "1fbce10ff45571633755eb27605fae880dc23a6f" user: "jenkins" date: "Fri May 25 02:35:53 UTC 2018" src_checksum: "7a9d63134f5103638fd0aa5cb7f15d0c" version_major: 3 version_minor: 0 2018-05-25 02:51:14,596 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.ServerManager(428): Registering regionserver=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,621 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0 2018-05-25 02:51:14,622 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:43302 2018-05-25 02:51:14,622 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-05-25 02:51:14,635 INFO [Thread-81] master.ServerManager(887): Finished waiting on RegionServer count=1; waited=201ms, expected min=1 server(s), max=1 server(s), master is running 2018-05-25 02:51:14,647 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-05-25 02:51:14,647 DEBUG [RS:0;asf911:36079] zookeeper.ZKUtil(355): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,648 WARN [RS:0;asf911:36079] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-05-25 02:51:14,649 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,658 INFO [RS:0;asf911:36079] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-05-25 02:51:14,669 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1811): logDir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,683 DEBUG [RS:0;asf911:36079] zookeeper.ZKUtil(355): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,690 DEBUG [RS:0;asf911:36079] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-05-25 02:51:14,700 INFO [RS:0;asf911:36079] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-05-25 02:51:14,736 INFO [RS:0;asf911:36079] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-05-25 02:51:14,743 INFO [RS:0;asf911:36079] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-05-25 02:51:14,745 INFO [RS:0;asf911:36079] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-05-25 02:51:14,762 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-05-25 02:51:14,763 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-05-25 02:51:14,763 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-05-25 02:51:14,763 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-05-25 02:51:14,764 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-05-25 02:51:14,764 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-05-25 02:51:14,764 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-05-25 02:51:14,765 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-05-25 02:51:14,765 DEBUG [RS:0;asf911:36079] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-05-25 02:51:14,798 INFO [SplitLogWorker-asf911:36079] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,36079,1527216672716 starting 2018-05-25 02:51:14,803 INFO [RS:0;asf911:36079] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-05-25 02:51:14,853 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,36079,1527216672716, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:36079, sessionid=0x163953502c90001 2018-05-25 02:51:14,853 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-05-25 02:51:14,853 DEBUG [RS:0;asf911:36079] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,853 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,36079,1527216672716' 2018-05-25 02:51:14,854 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-05-25 02:51:14,855 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-05-25 02:51:14,856 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-05-25 02:51:14,856 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-05-25 02:51:14,856 DEBUG [RS:0;asf911:36079] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:14,856 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,36079,1527216672716' 2018-05-25 02:51:14,856 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-05-25 02:51:14,857 DEBUG [RS:0;asf911:36079] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-05-25 02:51:14,857 DEBUG [RS:0;asf911:36079] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-05-25 02:51:14,858 INFO [RS:0;asf911:36079] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-05-25 02:51:14,858 INFO [RS:0;asf911:36079] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-05-25 02:51:14,916 DEBUG [Thread-81] procedure2.ProcedureExecutor(884): Stored pid=1, state=RUNNABLE:RECOVER_META_PREPARE; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-05-25 02:51:14,972 INFO [PEWorker-1] procedure.RecoverMetaProcedure(125): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-05-25 02:51:14,976 DEBUG [PEWorker-1] zookeeper.ZKUtil(614): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-05-25 02:51:14,981 DEBUG [PEWorker-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-05-25 02:51:15,048 DEBUG [PEWorker-1] zookeeper.ZKUtil(614): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-05-25 02:51:15,049 INFO [PEWorker-1] procedure.RecoverMetaProcedure(157): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-05-25 02:51:15,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-05-25 02:51:15,135 INFO [PEWorker-4] procedure.MasterProcedureScheduler(640): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 checking lock on 1588230740 2018-05-25 02:51:15,145 INFO [PEWorker-4] assignment.AssignProcedure(218): Starting pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-05-25 02:51:15,300 DEBUG [master/asf911:0] assignment.AssignmentManager(1774): Processing assignQueue; systemServersCount=1, allServersCount=1 2018-05-25 02:51:15,305 INFO [PEWorker-3] assignment.AssignProcedure(246): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:15,892 INFO [RS:0;asf911:36079] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C36079%2C1527216672716, suffix=, logDir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716, archiveDir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/oldWALs 2018-05-25 02:51:15,930 DEBUG [RS:0;asf911:36079] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-05-25 02:51:15,986 DEBUG [RS:0;asf911:36079] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:110) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:763) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:493) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:251) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:70) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:45) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:254) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2104) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1325) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1191) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:15,988 DEBUG [RS:0;asf911:36079] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-05-25 02:51:16,010 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:16,015 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42355,DS-6a595950-4d09-4fed-930f-9a9193bb9c36,DISK] 2018-05-25 02:51:16,047 INFO [RS:0;asf911:36079] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716/asf911.gq1.ygridcore.net%2C36079%2C1527216672716.1527216675910 2018-05-25 02:51:16,048 DEBUG [RS:0;asf911:36079] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42355,DS-6a595950-4d09-4fed-930f-9a9193bb9c36,DISK]] 2018-05-25 02:51:16,076 INFO [PEWorker-2] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,097 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-05-25 02:51:16,131 INFO [PEWorker-2] assignment.RegionTransitionProcedure(251): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,326 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(779): New admin connection to asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,336 INFO [RS-EventLoopGroup-3-6] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-05-25 02:51:16,337 INFO [RS-EventLoopGroup-3-6] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 37939 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "1fbce10ff45571633755eb27605fae880dc23a6f" user: "jenkins" date: "Fri May 25 02:35:53 UTC 2018" src_checksum: "7a9d63134f5103638fd0aa5cb7f15d0c" version_major: 3 version_minor: 0 2018-05-25 02:51:16,344 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open hbase:meta,,1.1588230740 2018-05-25 02:51:16,348 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-05-25 02:51:16,356 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C36079%2C1527216672716.meta, suffix=.meta, logDir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716, archiveDir=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/oldWALs 2018-05-25 02:51:16,375 DEBUG [RS-EventLoopGroup-3-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42355,DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786,DISK] 2018-05-25 02:51:16,383 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716/asf911.gq1.ygridcore.net%2C36079%2C1527216672716.meta.1527216676359.meta 2018-05-25 02:51:16,384 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42355,DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786,DISK]] 2018-05-25 02:51:16,385 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7088): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:16,415 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-05-25 02:51:16,434 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(8064): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-05-25 02:51:16,443 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-05-25 02:51:16,451 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-05-25 02:51:16,451 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:16,466 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/info 2018-05-25 02:51:16,466 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/info 2018-05-25 02:51:16,467 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:16,468 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:16,470 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:16,473 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/rep_barrier 2018-05-25 02:51:16,473 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/rep_barrier 2018-05-25 02:51:16,474 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:16,474 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:16,476 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:16,478 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/table 2018-05-25 02:51:16,479 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/table 2018-05-25 02:51:16,479 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:16,480 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:16,482 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:16,487 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(4469): Found 0 recovered edits file(s) under hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740 2018-05-25 02:51:16,491 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-05-25 02:51:16,493 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(975): Opened 1588230740; next sequenceid=2 2018-05-25 02:51:16,548 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2182): Post open deploy tasks for hbase:meta,,1.1588230740 2018-05-25 02:51:16,572 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report OPENED seqId=2, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,573 DEBUG [PEWorker-5] assignment.RegionTransitionProcedure(363): Finishing pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,574 INFO [PEWorker-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,575 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2206): Finished post open deploy task for hbase:meta,,1.1588230740 2018-05-25 02:51:16,577 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:16,596 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-05-25 02:51:16,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1634): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-05-25 02:51:16,808 INFO [PEWorker-5] procedure2.ProcedureExecutor(1265): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.5430sec 2018-05-25 02:51:16,989 INFO [PEWorker-6] procedure2.ProcedureExecutor(1265): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 2.1480sec 2018-05-25 02:51:16,990 INFO [Thread-81] master.HMaster(3597): hbase:meta deployed at=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:17,068 INFO [RS-EventLoopGroup-3-10] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-05-25 02:51:17,068 INFO [RS-EventLoopGroup-3-10] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 37987 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "1fbce10ff45571633755eb27605fae880dc23a6f" user: "jenkins" date: "Fri May 25 02:35:53 UTC 2018" src_checksum: "7a9d63134f5103638fd0aa5cb7f15d0c" version_major: 3 version_minor: 0 2018-05-25 02:51:17,102 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] assignment.AssignmentManager(1004): META REPORTED: rit=OPEN, location=asf911.gq1.ygridcore.net,36079,1527216672716, table=hbase:meta, region=1588230740 2018-05-25 02:51:17,102 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] assignment.AssignmentManager(1006): META REPORTED but no procedure found (complete?); set location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:17,233 INFO [Thread-81] master.MasterWalManager(189): Log folder hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/WALs/asf911.gq1.ygridcore.net,36079,1527216672716 belongs to an existing region server 2018-05-25 02:51:17,233 DEBUG [Thread-81] assignment.AssignmentManager(1209): Joining cluster... 2018-05-25 02:51:17,239 INFO [Thread-81] assignment.AssignmentManager(1220): Number of RegionServers=1 2018-05-25 02:51:17,239 INFO [Thread-81] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=1000, timestamp=1527216678239 2018-05-25 02:51:17,239 INFO [Thread-81] assignment.AssignmentManager(1227): Joined the cluster in 6msec, failover=true 2018-05-25 02:51:17,281 INFO [Thread-81] master.TableNamespaceManager(96): Namespace table not found. Creating... 2018-05-25 02:51:17,287 INFO [Thread-81] master.HMaster(1817): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-05-25 02:51:17,490 DEBUG [Thread-81] procedure2.ProcedureExecutor(884): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-05-25 02:51:17,711 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|RBW]]} size 476 2018-05-25 02:51:18,120 DEBUG [PEWorker-7] util.FSTableDescriptors(683): Wrote into hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-05-25 02:51:18,125 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6911): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/.tmp Table name == hbase:namespace 2018-05-25 02:51:18,145 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|FINALIZED]]} size 0 2018-05-25 02:51:18,147 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(827): Instantiated hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:18,147 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1514): Closing 92c5a46404e20e6e619d36e5e536e81f, disabling compactions & flushes 2018-05-25 02:51:18,148 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1554): Updates disabled for region hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:18,148 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1668): Closed hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:18,320 DEBUG [PEWorker-7] hbase.MetaTableAccessor(2145): Put {"totalColumns":2,"row":"hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":1527216678246},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1527216678246}]},"ts":1527216678246} 2018-05-25 02:51:18,434 INFO [PEWorker-7] hbase.MetaTableAccessor(1520): Added 1 regions to meta. 2018-05-25 02:51:18,510 DEBUG [PEWorker-7] hbase.MetaTableAccessor(2145): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1527216678503}]},"ts":1527216678503} 2018-05-25 02:51:18,535 INFO [PEWorker-7] hbase.MetaTableAccessor(1665): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2018-05-25 02:51:18,566 INFO [PEWorker-7] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716}] 2018-05-25 02:51:18,645 INFO [PEWorker-8] procedure.MasterProcedureScheduler(640): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716 checking lock on 92c5a46404e20e6e619d36e5e536e81f 2018-05-25 02:51:18,664 INFO [PEWorker-8] assignment.AssignProcedure(218): Starting pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36079,1527216672716; forceNewPlan=false, retain=false 2018-05-25 02:51:18,816 INFO [PEWorker-9] assignment.RegionStateStore(197): pid=4 updating hbase:meta row=92c5a46404e20e6e619d36e5e536e81f, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:18,827 INFO [PEWorker-9] assignment.RegionTransitionProcedure(251): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:18,980 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:18,996 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(7088): Opening region: {ENCODED => 92c5a46404e20e6e619d36e5e536e81f, NAME => 'hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:18,997 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 92c5a46404e20e6e619d36e5e536e81f 2018-05-25 02:51:18,998 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:19,004 DEBUG [StoreOpener-92c5a46404e20e6e619d36e5e536e81f-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/info 2018-05-25 02:51:19,004 DEBUG [StoreOpener-92c5a46404e20e6e619d36e5e536e81f-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/info 2018-05-25 02:51:19,006 INFO [StoreOpener-92c5a46404e20e6e619d36e5e536e81f-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:19,006 INFO [StoreOpener-92c5a46404e20e6e619d36e5e536e81f-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:19,008 INFO [StoreOpener-92c5a46404e20e6e619d36e5e536e81f-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:19,010 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(4469): Found 0 recovered edits file(s) under hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f 2018-05-25 02:51:19,029 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-05-25 02:51:19,029 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(975): Opened 92c5a46404e20e6e619d36e5e536e81f; next sequenceid=2 2018-05-25 02:51:19,037 INFO [PostOpenDeployTasks:92c5a46404e20e6e619d36e5e536e81f] regionserver.HRegionServer(2182): Post open deploy tasks for hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:19,040 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:19,041 DEBUG [PEWorker-10] assignment.RegionTransitionProcedure(363): Finishing pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:19,048 DEBUG [PostOpenDeployTasks:92c5a46404e20e6e619d36e5e536e81f] regionserver.HRegionServer(2206): Finished post open deploy task for hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:19,048 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. on asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:19,051 INFO [PEWorker-10] assignment.RegionStateStore(197): pid=4 updating hbase:meta row=92c5a46404e20e6e619d36e5e536e81f, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:19,211 INFO [PEWorker-10] procedure2.ProcedureExecutor(1634): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-05-25 02:51:19,211 INFO [PEWorker-10] procedure2.ProcedureExecutor(1265): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=92c5a46404e20e6e619d36e5e536e81f, target=asf911.gq1.ygridcore.net,36079,1527216672716 in 493msec 2018-05-25 02:51:19,211 DEBUG [PEWorker-11] hbase.MetaTableAccessor(2145): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1527216679211}]},"ts":1527216679211} 2018-05-25 02:51:19,219 INFO [PEWorker-11] hbase.MetaTableAccessor(1665): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2018-05-25 02:51:19,246 DEBUG [Thread-81] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-05-25 02:51:19,290 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-05-25 02:51:19,428 INFO [PEWorker-11] procedure2.ProcedureExecutor(1265): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 2.0350sec 2018-05-25 02:51:19,504 DEBUG [Thread-81] procedure2.ProcedureExecutor(884): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-05-25 02:51:19,738 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-05-25 02:51:19,964 INFO [PEWorker-13] procedure2.ProcedureExecutor(1265): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 566msec 2018-05-25 02:51:20,161 DEBUG [Thread-81] procedure2.ProcedureExecutor(884): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-05-25 02:51:20,363 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-05-25 02:51:20,556 INFO [PEWorker-12] procedure2.ProcedureExecutor(1265): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 481msec 2018-05-25 02:51:20,599 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-05-25 02:51:20,612 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-05-25 02:51:20,613 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-05-25 02:51:20,690 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-05-25 02:51:20,691 INFO [Thread-81] master.HMaster(957): Master has completed initialization 7.732sec 2018-05-25 02:51:20,694 INFO [Thread-81] quotas.MasterQuotaManager(90): Quota support disabled 2018-05-25 02:51:20,694 INFO [Thread-81] zookeeper.ZKWatcher(205): not a secure deployment, proceeding 2018-05-25 02:51:20,712 DEBUG [Thread-81] master.HMaster(1014): Balancer post startup initialization complete, took 0 seconds 2018-05-25 02:51:20,731 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-05-25 02:51:20,731 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-05-25 02:51:20,792 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x798248d5 to localhost:53584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-05-25 02:51:20,826 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e215752, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-05-25 02:51:20,860 INFO [RS-EventLoopGroup-3-11] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-05-25 02:51:20,861 INFO [RS-EventLoopGroup-3-11] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 38485 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "1fbce10ff45571633755eb27605fae880dc23a6f" user: "jenkins" date: "Fri May 25 02:35:53 UTC 2018" src_checksum: "7a9d63134f5103638fd0aa5cb7f15d0c" version_major: 3 version_minor: 0 2018-05-25 02:51:20,878 INFO [Time-limited test] hbase.HBaseTestingUtility(1045): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:20,936 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.TestAssignmentManagerMetrics#testRITAssignmentManagerMetrics Thread=208, OpenFileDescriptor=828, MaxFileDescriptor=60000, SystemLoadAverage=1309, ProcessCount=289, AvailableMemoryMB=1823 2018-05-25 02:51:20,956 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-05-25 02:51:20,956 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 52695 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "1fbce10ff45571633755eb27605fae880dc23a6f" user: "jenkins" date: "Fri May 25 02:35:53 UTC 2018" src_checksum: "7a9d63134f5103638fd0aa5cb7f15d0c" version_major: 3 version_minor: 0 2018-05-25 02:51:20,973 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.HMaster$3(1781): Client=jenkins//67.195.81.155 create 'testRITAssignmentManagerMetrics', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-05-25 02:51:21,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] procedure2.ProcedureExecutor(884): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRITAssignmentManagerMetrics 2018-05-25 02:51:21,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:21,281 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|RBW]]} size 504 2018-05-25 02:51:21,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:21,687 DEBUG [PEWorker-14] util.FSTableDescriptors(683): Wrote into hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/.tmp/data/default/testRITAssignmentManagerMetrics/.tabledesc/.tableinfo.0000000001 2018-05-25 02:51:21,692 INFO [RegionOpenAndInitThread-testRITAssignmentManagerMetrics-1] regionserver.HRegion(6911): creating HRegion testRITAssignmentManagerMetrics HTD == 'testRITAssignmentManagerMetrics', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/.tmp Table name == testRITAssignmentManagerMetrics 2018-05-25 02:51:21,722 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|FINALIZED]]} size 0 2018-05-25 02:51:21,724 DEBUG [RegionOpenAndInitThread-testRITAssignmentManagerMetrics-1] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:21,725 DEBUG [RegionOpenAndInitThread-testRITAssignmentManagerMetrics-1] regionserver.HRegion(1514): Closing 5869bebf1fdd1c2280de6863e839a6e3, disabling compactions & flushes 2018-05-25 02:51:21,725 DEBUG [RegionOpenAndInitThread-testRITAssignmentManagerMetrics-1] regionserver.HRegion(1554): Updates disabled for region testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:21,725 INFO [RegionOpenAndInitThread-testRITAssignmentManagerMetrics-1] regionserver.HRegion(1668): Closed testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:21,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:21,805 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2145): Put {"totalColumns":2,"row":"testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":1527216681804},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1527216681804}]},"ts":1527216681804} 2018-05-25 02:51:21,812 INFO [PEWorker-14] hbase.MetaTableAccessor(1520): Added 1 regions to meta. 2018-05-25 02:51:21,911 DEBUG [PEWorker-14] hbase.MetaTableAccessor(2145): Put {"totalColumns":1,"row":"testRITAssignmentManagerMetrics","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1527216681911}]},"ts":1527216681911} 2018-05-25 02:51:21,917 INFO [PEWorker-14] hbase.MetaTableAccessor(1665): Updated tableName=testRITAssignmentManagerMetrics, state=ENABLING in hbase:meta 2018-05-25 02:51:21,958 INFO [PEWorker-14] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716}] 2018-05-25 02:51:21,968 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-05-25 02:51:22,080 INFO [PEWorker-14] procedure.MasterProcedureScheduler(640): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716 checking lock on 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:22,089 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36079,1527216672716; forceNewPlan=false, retain=false 2018-05-25 02:51:22,244 WARN [ProcExecTimeout] assignment.AssignmentManager(1201): STUCK Region-In-Transition rit=OFFLINE, location=asf911.gq1.ygridcore.net,36079,1527216672716, table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:22,245 INFO [PEWorker-16] assignment.RegionStateStore(197): pid=8 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,254 INFO [PEWorker-16] assignment.RegionTransitionProcedure(251): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:22,407 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:22,418 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:22,418 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:22,419 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:22,429 DEBUG [StoreOpener-5869bebf1fdd1c2280de6863e839a6e3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/family 2018-05-25 02:51:22,430 DEBUG [StoreOpener-5869bebf1fdd1c2280de6863e839a6e3-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/family 2018-05-25 02:51:22,438 INFO [StoreOpener-5869bebf1fdd1c2280de6863e839a6e3-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-05-25 02:51:22,439 INFO [StoreOpener-5869bebf1fdd1c2280de6863e839a6e3-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-05-25 02:51:22,442 INFO [StoreOpener-5869bebf1fdd1c2280de6863e839a6e3-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-05-25 02:51:22,444 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4469): Found 0 recovered edits file(s) under hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:22,459 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-05-25 02:51:22,460 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(975): Opened 5869bebf1fdd1c2280de6863e839a6e3; next sequenceid=2 2018-05-25 02:51:22,466 INFO [PostOpenDeployTasks:5869bebf1fdd1c2280de6863e839a6e3] regionserver.HRegionServer(2182): Post open deploy tasks for testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,470 DEBUG [PEWorker-1] assignment.RegionTransitionProcedure(363): Finishing pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,470 DEBUG [PostOpenDeployTasks:5869bebf1fdd1c2280de6863e839a6e3] regionserver.HRegionServer(2206): Finished post open deploy task for testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:22,471 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. on asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,477 INFO [PEWorker-1] assignment.RegionStateStore(197): pid=8 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:22,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1634): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testRITAssignmentManagerMetrics; resume parent processing. 2018-05-25 02:51:22,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1265): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716 in 527msec 2018-05-25 02:51:22,597 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2145): Put {"totalColumns":1,"row":"testRITAssignmentManagerMetrics","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1527216682597}]},"ts":1527216682597} 2018-05-25 02:51:22,612 INFO [PEWorker-4] hbase.MetaTableAccessor(1665): Updated tableName=testRITAssignmentManagerMetrics, state=ENABLED in hbase:meta 2018-05-25 02:51:22,761 INFO [PEWorker-4] procedure2.ProcedureExecutor(1265): Finished pid=7, state=SUCCESS; CreateTableProcedure table=testRITAssignmentManagerMetrics in 1.7210sec 2018-05-25 02:51:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.MasterRpcServices(1141): Checking to see if procedure is done pid=7 2018-05-25 02:51:23,301 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3645): Operation: CREATE, Table Name: default:testRITAssignmentManagerMetrics completed 2018-05-25 02:51:23,302 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3284): Waiting until all regions of table testRITAssignmentManagerMetrics get assigned. Timeout = 5000ms 2018-05-25 02:51:23,306 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [5,000] milli-secs(wait.for.ratio=[1]) 2018-05-25 02:51:23,317 INFO [Time-limited test] hbase.HBaseTestingUtility(3336): All regions for table testRITAssignmentManagerMetrics assigned to meta. Checking AM states. 2018-05-25 02:51:23,318 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [5,000] milli-secs(wait.for.ratio=[1]) 2018-05-25 02:51:23,318 INFO [Time-limited test] hbase.HBaseTestingUtility(3356): All regions for table testRITAssignmentManagerMetrics assigned. 2018-05-25 02:51:26,366 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.HMaster(2023): hdfs:/foo.jar 2018-05-25 02:51:26,367 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] master.HMaster$9(2389): Client=jenkins//67.195.81.155 modify testRITAssignmentManagerMetrics 2018-05-25 02:51:26,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] procedure2.ProcedureExecutor(884): Stored pid=9, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRITAssignmentManagerMetrics 2018-05-25 02:51:26,732 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741836_1012{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 582 2018-05-25 02:51:26,735 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-05-25 02:51:27,138 DEBUG [PEWorker-3] util.FSTableDescriptors(683): Wrote into hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/.tabledesc/.tableinfo.0000000002 2018-05-25 02:51:27,147 INFO [IPC Server handler 5 on 43302] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741834_1010 127.0.0.1:42355 2018-05-25 02:51:27,151 DEBUG [PEWorker-3] util.FSTableDescriptors(629): Deleted hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/.tabledesc/.tableinfo.0000000001 2018-05-25 02:51:27,151 INFO [PEWorker-3] util.FSTableDescriptors(594): Updated tableinfo=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/.tabledesc/.tableinfo.0000000002 2018-05-25 02:51:27,554 INFO [PEWorker-3] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=5869bebf1fdd1c2280de6863e839a6e3, source=asf911.gq1.ygridcore.net,36079,1527216672716, destination=asf911.gq1.ygridcore.net,36079,1527216672716}] 2018-05-25 02:51:27,676 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@308419d7] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:42355 to delete [blk_1073741834_1010] 2018-05-25 02:51:27,692 INFO [PEWorker-2] procedure.MasterProcedureScheduler(640): pid=10, ppid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=5869bebf1fdd1c2280de6863e839a6e3, source=asf911.gq1.ygridcore.net,36079,1527216672716, destination=asf911.gq1.ygridcore.net,36079,1527216672716 checking lock on 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:27,693 INFO [PEWorker-2] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716}] 2018-05-25 02:51:27,773 INFO [PEWorker-2] procedure.MasterProcedureScheduler(640): pid=11, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716 checking lock on 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:27,773 INFO [PEWorker-2] assignment.RegionStateStore(197): pid=11 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:27,783 INFO [PEWorker-2] assignment.RegionTransitionProcedure(251): Dispatch pid=11, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716; rit=CLOSING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:27,924 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-05-25 02:51:27,939 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1604): Close 5869bebf1fdd1c2280de6863e839a6e3, moving to asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:27,944 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1514): Closing 5869bebf1fdd1c2280de6863e839a6e3, disabling compactions & flushes 2018-05-25 02:51:27,944 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1554): Updates disabled for region testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:27,945 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2554): Flushing 1/1 column families, dataSize=43 B heapSize=120 B 2018-05-25 02:51:28,073 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741837_1013{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 4686 2018-05-25 02:51:28,247 WARN [ProcExecTimeout] assignment.AssignmentManager(1201): STUCK Region-In-Transition rit=CLOSING, location=asf911.gq1.ygridcore.net,36079,1527216672716, table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:28,475 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=43 B at sequenceid=5 (bloomFilter=false), to=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/.tmp/family/b9b3450c6abc4420b0c8d6bd8957890d 2018-05-25 02:51:28,578 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/.tmp/family/b9b3450c6abc4420b0c8d6bd8957890d as hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/family/b9b3450c6abc4420b0c8d6bd8957890d 2018-05-25 02:51:28,593 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/family/b9b3450c6abc4420b0c8d6bd8957890d, entries=1, sequenceid=5, filesize=4.6 K 2018-05-25 02:51:28,605 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2763): Finished flush of dataSize ~43 B/43, heapSize ~336 B/336, currentSize=0 B/0 for 5869bebf1fdd1c2280de6863e839a6e3 in 659ms, sequenceid=5, compaction requested=false 2018-05-25 02:51:28,624 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/default/testRITAssignmentManagerMetrics/5869bebf1fdd1c2280de6863e839a6e3/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2018-05-25 02:51:28,634 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1668): Closed testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:28,635 WARN [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionServer(3418): Not adding moved region record: 5869bebf1fdd1c2280de6863e839a6e3 to self. 2018-05-25 02:51:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report CLOSED seqId=-1, pid=11, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716; rit=CLOSING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:28,639 DEBUG [PEWorker-6] assignment.RegionTransitionProcedure(363): Finishing pid=11, ppid=10, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716; rit=CLOSING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:28,639 INFO [PEWorker-6] assignment.RegionStateStore(197): pid=11 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=CLOSED 2018-05-25 02:51:28,639 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:28,806 INFO [PEWorker-6] procedure2.ProcedureExecutor(1634): Finished subprocedure(s) of pid=10, ppid=9, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=5869bebf1fdd1c2280de6863e839a6e3, source=asf911.gq1.ygridcore.net,36079,1527216672716, destination=asf911.gq1.ygridcore.net,36079,1527216672716; resume parent processing. 2018-05-25 02:51:28,806 INFO [PEWorker-7] procedure2.ProcedureExecutor(1515): Initialized subprocedures=[{pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716}] 2018-05-25 02:51:28,806 INFO [PEWorker-6] procedure2.ProcedureExecutor(1265): Finished pid=11, ppid=10, state=SUCCESS; UnassignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, server=asf911.gq1.ygridcore.net,36079,1527216672716 in 951msec 2018-05-25 02:51:28,860 INFO [Time-limited test] master.TestAssignmentManagerMetrics(150): Expected error java.net.SocketTimeoutException: callTimeout=2500, callDuration=2616: Call to asf911.gq1.ygridcore.net/67.195.81.155:46610 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=19, waitTime=2505, rpcTimeout=2498 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:158) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3061) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3053) at org.apache.hadoop.hbase.client.HBaseAdmin.modifyTableAsync(HBaseAdmin.java:349) at org.apache.hadoop.hbase.client.HBaseAdmin.modifyTable(HBaseAdmin.java:344) at org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.testRITAssignmentManagerMetrics(TestAssignmentManagerMetrics.java:144) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:46610 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=19, waitTime=2505, rpcTimeout=2498 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96) at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:199) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466) ... 1 more Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=19, waitTime=2505, rpcTimeout=2498 at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:200) ... 4 more 2018-05-25 02:51:28,863 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [900,000] milli-secs(wait.for.ratio=[1]) 2018-05-25 02:51:28,897 INFO [PEWorker-7] procedure.MasterProcedureScheduler(640): pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716 checking lock on 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:28,901 INFO [PEWorker-7] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OFFLINE, location=asf911.gq1.ygridcore.net,36079,1527216672716; forceNewPlan=false, retain=false 2018-05-25 02:51:29,052 INFO [PEWorker-9] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:29,057 INFO [PEWorker-9] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,210 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:29,218 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:29,223 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:29,227 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,228 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:29,228 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:29,236 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,238 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=1 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3, target=asf911.gq1.ygridcore.net,36079,1527216672716; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,242 INFO [PEWorker-10] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:29,393 INFO [PEWorker-11] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:29,398 INFO [PEWorker-11] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,559 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:29,582 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:29,587 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:29,607 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,608 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:29,608 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:29,622 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,629 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=2 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,635 INFO [PEWorker-13] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:29,787 INFO [PEWorker-12] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:29,792 INFO [PEWorker-12] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,944 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:29,951 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:29,954 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:29,957 ERROR [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,958 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:29,958 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:29,963 ERROR [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,966 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=3 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:29,969 INFO [PEWorker-15] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:30,120 INFO [PEWorker-14] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:30,125 INFO [PEWorker-14] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,248 WARN [ProcExecTimeout] assignment.AssignmentManager(1201): STUCK Region-In-Transition rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716, table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:30,277 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:30,284 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:30,287 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:30,291 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,292 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:30,293 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:30,298 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,301 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=4 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,303 INFO [PEWorker-16] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:30,454 INFO [PEWorker-1] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:30,459 INFO [PEWorker-1] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,612 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:30,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:30,623 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:30,627 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,628 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:30,628 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:30,634 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,637 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=5 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,640 INFO [PEWorker-4] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:30,791 INFO [PEWorker-3] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:30,796 INFO [PEWorker-3] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,949 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:30,956 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:30,960 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:30,963 ERROR [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,964 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:30,964 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:30,970 ERROR [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:30,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,972 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=6 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:30,975 INFO [PEWorker-5] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:31,126 INFO [PEWorker-2] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:31,132 INFO [PEWorker-2] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,248 WARN [ProcExecTimeout] assignment.AssignmentManager(1201): STUCK Region-In-Transition rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716, table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:31,285 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:31,291 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:31,295 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:31,299 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:31,300 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:31,300 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:31,307 ERROR [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:31,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,309 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=7 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,313 INFO [PEWorker-6] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:31,464 INFO [PEWorker-8] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:31,469 INFO [PEWorker-8] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,621 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:31,628 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7088): Opening region: {ENCODED => 5869bebf1fdd1c2280de6863e839a6e3, NAME => 'testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.', STARTKEY => '', ENDKEY => ''} 2018-05-25 02:51:31,632 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] coprocessor.CoprocessorHost(200): Loading coprocessor class com.foo.FooRegionObserver with path hdfs:/foo.jar and priority 1001 2018-05-25 02:51:31,637 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.RegionCoprocessorHost(401): Failed to load coprocessor com.foo.FooRegionObserver java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:211) at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:388) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:275) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:818) at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:709) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6890) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7090) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:31,637 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRITAssignmentManagerMetrics 5869bebf1fdd1c2280de6863e839a6e3 2018-05-25 02:51:31,638 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-05-25 02:51:31,644 ERROR [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(302): Failed open of region=testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3., starting to roll back the global memstore size. java.io.FileNotFoundException: hdfs:/foo.jar at org.apache.hadoop.hbase.util.CoprocessorClassLoader.init(CoprocessorClassLoader.java:171) at org.apache.hadoop.hbase.util.CoprocessorClassLoader.getClassLoader(CoprocessorClassLoader.java:270) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:348) at org.apache.hadoop.hbase.regionserver.HRegion.checkClassLoading(HRegion.java:7210) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7132) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7091) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7062) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7018) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6969) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:109) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-05-25 02:51:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.RegionTransitionProcedure(273): Received report FAILED_OPEN seqId=-1, pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,647 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=46610] assignment.AssignProcedure(338): Retry=8 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,650 INFO [PEWorker-7] assignment.AssignProcedure(218): Starting pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-05-25 02:51:31,801 INFO [PEWorker-9] assignment.RegionStateStore(197): pid=12 updating hbase:meta row=5869bebf1fdd1c2280de6863e839a6e3, regionState=OPENING 2018-05-25 02:51:31,806 INFO [PEWorker-9] assignment.RegionTransitionProcedure(251): Dispatch pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,920 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.TestAssignmentManagerMetrics#testRITAssignmentManagerMetrics Thread=221 (was 208) Potentially hanging thread: RSProcedureDispatcher-pool3-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf911:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf911:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RSProcedureDispatcher-pool3-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf911:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf911:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-347844774_23 at /127.0.0.1:42434 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=830 (was 828) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=1245 (was 1309), ProcessCount=289 (was 289), AvailableMemoryMB=1499 (was 1823) 2018-05-25 02:51:31,921 INFO [Time-limited test] master.TestAssignmentManagerMetrics(104): AFTER org.apache.hadoop.hbase.HBaseTestingUtility@7d7a80a1 <= IS THIS NULL? 2018-05-25 02:51:31,921 INFO [Time-limited test] hbase.HBaseTestingUtility(1094): Shutting down minicluster 2018-05-25 02:51:31,922 INFO [Time-limited test] client.ConnectionImplementation(1754): Closing master protocol: MasterService 2018-05-25 02:51:31,922 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x798248d5 to localhost:53584 2018-05-25 02:51:31,922 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-05-25 02:51:31,924 DEBUG [Time-limited test] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-05-25 02:51:31,924 INFO [Time-limited test] master.ServerManager(958): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:31,959 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] regionserver.RSRpcServices(1980): Open testRITAssignmentManagerMetrics,,1527216680969.5869bebf1fdd1c2280de6863e839a6e3. 2018-05-25 02:51:31,963 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-05-25 02:51:31,963 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-05-25 02:51:31,963 INFO [Time-limited test] procedure2.ProcedureExecutor(588): Stopping 2018-05-25 02:51:31,964 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x4c970a31 to localhost:53584 2018-05-25 02:51:31,964 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-05-25 02:51:31,965 INFO [Time-limited test] regionserver.HRegionServer(2143): ***** STOPPING region server 'asf911.gq1.ygridcore.net,36079,1527216672716' ***** 2018-05-25 02:51:31,965 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-05-25 02:51:31,965 INFO [Time-limited test] regionserver.HRegionServer(2157): STOPPED: Shutdown requested 2018-05-25 02:51:31,966 WARN [RSProcedureDispatcher-pool3-t13] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(209): Failed dispatch to server=asf911.gq1.ygridcore.net,36079,1527216672716 try=0 java.io.IOException: Call to asf911.gq1.ygridcore.net/67.195.81.155:36079 failed on local exception: java.io.IOException: Connection closed at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:202) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:210) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Connection closed ... 25 more 2018-05-25 02:51:31,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-05-25 02:51:31,970 WARN [RSProcedureDispatcher-pool3-t13] assignment.RegionTransitionProcedure(225): Remote call failed pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716; exception=Call to asf911.gq1.ygridcore.net/67.195.81.155:36079 failed on local exception: java.io.IOException: Connection closed 2018-05-25 02:51:31,970 INFO [RSProcedureDispatcher-pool3-t13] assignment.AssignProcedure(338): Retry=9 of max=10; pid=12, ppid=10, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testRITAssignmentManagerMetrics, region=5869bebf1fdd1c2280de6863e839a6e3; rit=OPENING, location=asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,966 INFO [RS:0;asf911:36079] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-05-25 02:51:31,971 INFO [RS:0;asf911:36079] regionserver.HeapMemoryManager(221): Stopping 2018-05-25 02:51:31,975 INFO [RS:0;asf911:36079] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-05-25 02:51:31,977 INFO [RS:0;asf911:36079] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-05-25 02:51:31,975 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-05-25 02:51:31,976 INFO [SplitLogWorker-asf911:36079] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-05-25 02:51:31,978 INFO [SplitLogWorker-asf911:36079] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,36079,1527216672716 exiting 2018-05-25 02:51:31,975 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-05-25 02:51:31,979 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1514): Closing 92c5a46404e20e6e619d36e5e536e81f, disabling compactions & flushes 2018-05-25 02:51:31,978 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:31,979 DEBUG [RS:0;asf911:36079] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-05-25 02:51:31,981 INFO [RS:0;asf911:36079] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x5bdfe4e5 to localhost:53584 2018-05-25 02:51:31,981 DEBUG [RS:0;asf911:36079] ipc.AbstractRpcClient(483): Stopping rpc client 2018-05-25 02:51:31,981 INFO [RS:0;asf911:36079] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-05-25 02:51:31,981 INFO [RS:0;asf911:36079] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-05-25 02:51:31,981 INFO [RS:0;asf911:36079] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-05-25 02:51:31,982 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1399): Waiting on 2 regions to close 2018-05-25 02:51:31,983 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1514): Closing 1588230740, disabling compactions & flushes 2018-05-25 02:51:31,983 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1554): Updates disabled for region hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:31,983 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1554): Updates disabled for region hbase:meta,,1.1588230740 2018-05-25 02:51:31,984 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2554): Flushing 1/1 column families, dataSize=78 B heapSize=232 B 2018-05-25 02:51:31,984 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(2554): Flushing 3/3 column families, dataSize=6.09 KB heapSize=9.57 KB 2018-05-25 02:51:31,985 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1403): Online Regions={1588230740=hbase:meta,,1.1588230740, 92c5a46404e20e6e619d36e5e536e81f=hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f.} 2018-05-25 02:51:32,137 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741838_1014{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 7568 2018-05-25 02:51:32,146 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|RBW]]} size 0 2018-05-25 02:51:32,152 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/.tmp/info/dcefedac85fb43779d652a1552f5d318 2018-05-25 02:51:32,175 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/.tmp/info/dcefedac85fb43779d652a1552f5d318 as hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/info/dcefedac85fb43779d652a1552f5d318 2018-05-25 02:51:32,195 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/info/dcefedac85fb43779d652a1552f5d318, entries=2, sequenceid=6, filesize=4.8 K 2018-05-25 02:51:32,199 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2763): Finished flush of dataSize ~78 B/78, heapSize ~448 B/448, currentSize=0 B/0 for 92c5a46404e20e6e619d36e5e536e81f in 215ms, sequenceid=6, compaction requested=false 2018-05-25 02:51:32,219 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/namespace/92c5a46404e20e6e619d36e5e536e81f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2018-05-25 02:51:32,223 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1668): Closed hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:32,223 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed hbase:namespace,,1527216677282.92c5a46404e20e6e619d36e5e536e81f. 2018-05-25 02:51:32,540 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.87 KB at sequenceid=25 (bloomFilter=false), to=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/.tmp/info/7d93dced6b8647a590a33277fbde7338 2018-05-25 02:51:32,641 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 4896 2018-05-25 02:51:32,769 INFO [regionserver/asf911:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-05-25 02:51:32,770 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-05-25 02:51:32,986 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1399): Waiting on 1 regions to close 2018-05-25 02:51:32,986 DEBUG [RS:0;asf911:36079] regionserver.HRegionServer(1403): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-05-25 02:51:33,044 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=220 B at sequenceid=25 (bloomFilter=false), to=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/.tmp/table/ad488e98e8a142fd94f4515e55e1391a 2018-05-25 02:51:33,065 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/.tmp/info/7d93dced6b8647a590a33277fbde7338 as hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/info/7d93dced6b8647a590a33277fbde7338 2018-05-25 02:51:33,081 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/info/7d93dced6b8647a590a33277fbde7338, entries=21, sequenceid=25, filesize=7.4 K 2018-05-25 02:51:33,085 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/.tmp/table/ad488e98e8a142fd94f4515e55e1391a as hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/table/ad488e98e8a142fd94f4515e55e1391a 2018-05-25 02:51:33,099 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/table/ad488e98e8a142fd94f4515e55e1391a, entries=4, sequenceid=25, filesize=4.8 K 2018-05-25 02:51:33,102 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(2763): Finished flush of dataSize ~6.09 KB/6234, heapSize ~9.99 KB/10232, currentSize=0 B/0 for 1588230740 in 1118ms, sequenceid=25, compaction requested=false 2018-05-25 02:51:33,119 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] wal.WALSplitter(677): Wrote file=hdfs://localhost:43302/user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/data/hbase/meta/1588230740/recovered.edits/28.seqid, newMaxSeqId=28, maxSeqId=1 2018-05-25 02:51:33,151 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-05-25 02:51:33,154 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1668): Closed hbase:meta,,1.1588230740 2018-05-25 02:51:33,155 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-05-25 02:51:33,186 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,36079,1527216672716; all regions closed. 2018-05-25 02:51:33,195 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|RBW]]} size 0 2018-05-25 02:51:33,206 DEBUG [RS:0;asf911:36079] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/oldWALs 2018-05-25 02:51:33,206 INFO [RS:0;asf911:36079] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C36079%2C1527216672716.meta:.meta(num 1527216676359) 2018-05-25 02:51:33,232 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a9d3a9b2-b4aa-4fa8-8760-011beb254786:NORMAL:127.0.0.1:42355|RBW]]} size 0 2018-05-25 02:51:33,237 DEBUG [RS:0;asf911:36079] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/6c6f6890-e8ad-4f56-9e8e-dc17b0aec4c0/oldWALs 2018-05-25 02:51:33,237 INFO [RS:0;asf911:36079] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C36079%2C1527216672716:(num 1527216675910) 2018-05-25 02:51:33,238 DEBUG [RS:0;asf911:36079] ipc.AbstractRpcClient(483): Stopping rpc client 2018-05-25 02:51:33,238 INFO [RS:0;asf911:36079] regionserver.Leases(149): Closed leases 2018-05-25 02:51:33,238 INFO [RS:0;asf911:36079] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,36079,1527216672716 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-05-25 02:51:33,239 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-05-25 02:51:33,243 INFO [RS:0;asf911:36079] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:36079 2018-05-25 02:51:33,265 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:33,265 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,36079,1527216672716 2018-05-25 02:51:33,265 INFO [Time-limited test-EventThread] master.RegionServerTracker(132): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,36079,1527216672716] 2018-05-25 02:51:33,265 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:36079-0x163953502c90001, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-05-25 02:51:33,267 INFO [Time-limited test-EventThread] master.ServerManager(589): Cluster shutdown set; asf911.gq1.ygridcore.net,36079,1527216672716 expired; onlineServers=0 2018-05-25 02:51:33,267 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2143): ***** STOPPING region server 'asf911.gq1.ygridcore.net,46610,1527216670401' ***** 2018-05-25 02:51:33,267 INFO [Time-limited test-EventThread] regionserver.HRegionServer(2157): STOPPED: Cluster shutdown set; onlineServer=0 2018-05-25 02:51:33,268 DEBUG [M:0;asf911:46610] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a628802, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-05-25 02:51:33,268 INFO [M:0;asf911:46610] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,46610,1527216670401 2018-05-25 02:51:33,268 DEBUG [M:0;asf911:46610] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-05-25 02:51:33,268 INFO [M:0;asf911:46610] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,46610,1527216670401; all regions closed. 2018-05-25 02:51:33,269 DEBUG [M:0;asf911:46610] ipc.AbstractRpcClient(483): Stopping rpc client 2018-05-25 02:51:33,269 INFO [M:0;asf911:46610] hbase.ChoreService(327): Chore service for: master/asf911:0 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,46610,1527216670401-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46610,1527216670401-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46610,1527216670401-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:46610 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationBarrierCleaner Period: 43200000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46610,1527216670401-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,46610,1527216670401-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS]] on shutdown 2018-05-25 02:51:33,270 INFO [M:0;asf911:46610] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-05-25 02:51:33,270 INFO [M:0;asf911:46610] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-05-25 02:51:33,270 DEBUG [M:0;asf911:46610] master.HMaster(1233): Stopping service threads 2018-05-25 02:51:33,270 DEBUG [Thread-81-HFileCleaner.large.0-1527216674432] cleaner.HFileCleaner(253): Exit Thread[Thread-81-HFileCleaner.large.0-1527216674432,5,FailOnTimeoutGroup] 2018-05-25 02:51:33,271 DEBUG [OldWALsCleaner-0] cleaner.LogCleaner(176): Exiting cleaner. 2018-05-25 02:51:33,270 DEBUG [OldWALsCleaner-1] cleaner.LogCleaner(176): Exiting cleaner. 2018-05-25 02:51:33,270 DEBUG [Thread-81-HFileCleaner.small.0-1527216674432] cleaner.HFileCleaner(253): Exit Thread[Thread-81-HFileCleaner.small.0-1527216674432,5,FailOnTimeoutGroup] 2018-05-25 02:51:33,298 INFO [RS:0;asf911:36079] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,36079,1527216672716; zookeeper connection closed. 2018-05-25 02:51:33,299 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1244f7e9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1244f7e9 2018-05-25 02:51:33,301 INFO [Time-limited test] util.JVMClusterUtil(326): Shutdown of 1 master(s) and 1 regionserver(s) complete 2018-05-25 02:51:33,324 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-05-25 02:51:33,324 DEBUG [M:0;asf911:46610] zookeeper.RecoverableZooKeeper(176): Node /hbase/master already deleted, retry=false 2018-05-25 02:51:33,324 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-05-25 02:51:33,324 DEBUG [M:0;asf911:46610] master.ActiveMasterManager(280): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = NoNode for /hbase/master 2018-05-25 02:51:33,324 INFO [M:0;asf911:46610] assignment.AssignmentManager(234): Stopping assignment manager 2018-05-25 02:51:33,325 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:46610-0x163953502c90000, quorum=localhost:53584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-05-25 02:51:33,325 INFO [M:0;asf911:46610] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-05-25 02:51:33,327 INFO [M:0;asf911:46610] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-05-25 02:51:33,361 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42355 is added to blk_1073741829_1005{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6a595950-4d09-4fed-930f-9a9193bb9c36:NORMAL:127.0.0.1:42355|RBW]]} size 36245 2018-05-25 02:51:33,763 INFO [M:0;asf911:46610] hbase.ChoreService(327): Chore service for: master/asf911:0.splitLogManager. had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-05-25 02:51:33,764 INFO [M:0;asf911:46610] flush.MasterFlushTableProcedureManager(81): stop: server shutting down. 2018-05-25 02:51:33,765 INFO [M:0;asf911:46610] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:46610 2018-05-25 02:51:33,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=46610] ipc.CallRunner(142): callId: 19 service: MasterService methodName: ModifyTable size: 671 connection: 67.195.81.155:52695 deadline: 1527216688850, exception=java.io.InterruptedIOException 2018-05-25 02:51:33,779 DEBUG [M:0;asf911:46610] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf911.gq1.ygridcore.net,46610,1527216670401 already deleted, retry=false 2018-05-25 02:51:33,813 INFO [M:0;asf911:46610] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,46610,1527216670401; zookeeper connection closed. 2018-05-25 02:51:33,815 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-05-25 02:51:33,878 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-05-25 02:51:33,984 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:43302] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-05-25 02:51:34,000 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/36b0bd43-4a33-47ba-89ea-de32e9b5ae63/cluster_68792632-4a6a-43e0-aded-e7666e982fbf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:43302] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-2108902099-67.195.81.155-1527216663755 (Datanode Uuid 1621ce37-98d3-4d8e-87e2-8c5d539f68c6) service to localhost/127.0.0.1:43302 2018-05-25 02:51:34,044 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-05-25 02:51:34,134 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-05-25 02:51:34,172 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-05-25 02:51:34,176 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(324): Shutdown MiniZK cluster with all ZK servers 2018-05-25 02:51:34,230 INFO [Time-limited test] hbase.HBaseTestingUtility(1105): Minicluster is down 2018-05-25 02:51:34,277 DEBUG [Thread-81-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner-0x163953502c90004, quorum=localhost:53584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-05-25 02:51:34,278 DEBUG [Thread-81-EventThread] zookeeper.ZKWatcher(546): replicationLogCleaner-0x163953502c90004, quorum=localhost:53584, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-05-25 02:51:34,322 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@18f25ec9 2018-05-25 02:51:34,323 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-05-25 02:51:34,328 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.