2018-10-17 21:46:52,934 DEBUG [main] hbase.HBaseTestingUtility(350): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d 2018-10-17 21:46:52,968 INFO [Time-limited test] hbase.HBaseTestingUtility(1030): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2018-10-17 21:46:52,969 INFO [Time-limited test] hbase.HBaseZKTestingUtility(85): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b, deleteOnExit=true 2018-10-17 21:46:52,969 INFO [Time-limited test] hbase.HBaseTestingUtility(1044): STARTING DFS 2018-10-17 21:46:52,971 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cache_data in system properties and HBase conf 2018-10-17 21:46:52,972 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/hadoop_tmp in system properties and HBase conf 2018-10-17 21:46:52,972 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/hadoop_logs in system properties and HBase conf 2018-10-17 21:46:52,973 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/mapred_local in system properties and HBase conf 2018-10-17 21:46:52,974 INFO [Time-limited test] hbase.HBaseTestingUtility(752): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/mapred_temp in system properties and HBase conf 2018-10-17 21:46:52,994 INFO [Time-limited test] hbase.HBaseTestingUtility(743): read short circuit is OFF 2018-10-17 21:46:53,349 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-10-17 21:46:54,640 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-10-17 21:46:59,213 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-10-17 21:46:59,708 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-10-17 21:46:59,973 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-10-17 21:47:00,056 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/local-repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_44940_hdfs____.ea0msh/webapp 2018-10-17 21:47:00,564 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44940 2018-10-17 21:47:02,994 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-10-17 21:47:03,052 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/local-repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_56967_datanode____.gxzb4w/webapp 2018-10-17 21:47:03,363 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56967 2018-10-17 21:47:10,262 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:34726] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-10-17 21:47:10,714 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1ee0941cefdad2: from storage DS-cfcf6b71-0581-475e-be94-80a5198ae438 node DatanodeRegistration(127.0.0.1:46034, datanodeUuid=ea0c7941-9642-499f-935e-afb50ff6339a, infoPort=47656, infoSecurePort=0, ipcPort=52766, storageInfo=lv=-56;cid=testClusterID;nsid=1023355250;c=0), blocks: 0, hasStaleStorage: true, processing time: 38 msecs 2018-10-17 21:47:10,718 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1ee0941cefdad2: from storage DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef node DatanodeRegistration(127.0.0.1:46034, datanodeUuid=ea0c7941-9642-499f-935e-afb50ff6339a, infoPort=47656, infoSecurePort=0, ipcPort=52766, storageInfo=lv=-56;cid=testClusterID;nsid=1023355250;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-10-17 21:47:10,730 DEBUG [Time-limited test] hbase.HBaseTestingUtility(678): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d 2018-10-17 21:47:10,968 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-10-17 21:47:11,034 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran successful 'stat' on client port=63733 2018-10-17 21:47:11,083 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:11,089 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:12,203 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 7 2018-10-17 21:47:12,626 INFO [Time-limited test] util.FSUtils(515): Created version file at hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae with version=8 2018-10-17 21:47:12,627 INFO [Time-limited test] hbase.HBaseTestingUtility(1344): Setting hbase.fs.tmp.dir to hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase-staging 2018-10-17 21:47:13,320 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-10-17 21:47:13,938 INFO [Time-limited test] client.ConnectionUtils(122): master/asf904:0 server-side Connection retries=45 2018-10-17 21:47:13,993 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-10-17 21:47:13,995 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-10-17 21:47:13,996 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-10-17 21:47:14,335 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientService, hbase.pb.AdminService 2018-10-17 21:47:14,680 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-10-17 21:47:14,928 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /67.195.81.148:41507 2018-10-17 21:47:14,961 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: CacheConfig:disabled 2018-10-17 21:47:14,962 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: CacheConfig:disabled 2018-10-17 21:47:14,974 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-10-17 21:47:14,977 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:14,991 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:15,801 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:41507 connecting to ZooKeeper ensemble=localhost:63733 2018-10-17 21:47:15,962 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:415070x0, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-17 21:47:15,964 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): master:41507-0x16683ff2ebf0000 connected 2018-10-17 21:47:16,105 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-10-17 21:47:16,107 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-17 21:47:16,150 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41507 2018-10-17 21:47:16,218 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=41507 2018-10-17 21:47:16,242 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41507 2018-10-17 21:47:16,271 INFO [Time-limited test] master.HMaster(505): hbase.rootdir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae, hbase.cluster.distributed=false 2018-10-17 21:47:16,757 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf904:0 server-side Connection retries=45 2018-10-17 21:47:16,758 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-10-17 21:47:16,774 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-10-17 21:47:16,775 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-10-17 21:47:16,781 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-10-17 21:47:16,791 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-10-17 21:47:16,822 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /67.195.81.148:47370 2018-10-17 21:47:16,823 INFO [Time-limited test] hfile.CacheConfig(575): Allocating onheap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-10-17 21:47:16,856 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:16,858 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:16,865 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:16,887 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:16,896 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:47370 connecting to ZooKeeper ensemble=localhost:63733 2018-10-17 21:47:17,034 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:473700x0, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-17 21:47:17,062 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:47370-0x16683ff2ebf0001 connected 2018-10-17 21:47:17,062 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-10-17 21:47:17,074 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-17 21:47:17,106 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=47370 2018-10-17 21:47:17,158 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=47370 2018-10-17 21:47:17,178 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=47370 2018-10-17 21:47:17,222 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(2288): Adding backup master ZNode /hbase/backup-masters/asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:17,278 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:17,414 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-10-17 21:47:17,418 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-10-17 21:47:17,438 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-10-17 21:47:17,447 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-10-17 21:47:17,454 INFO [master/asf904:0:becomeActiveMaster] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf904.gq1.ygridcore.net,41507,1539812832861 from backup master directory 2018-10-17 21:47:17,466 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:17,468 WARN [master/asf904:0:becomeActiveMaster] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-10-17 21:47:17,468 INFO [master/asf904:0:becomeActiveMaster] master.ActiveMasterManager(181): Registered as active master=asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:17,486 INFO [master/asf904:0:becomeActiveMaster] regionserver.ChunkCreator(498): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-10-17 21:47:17,498 INFO [master/asf904:0:becomeActiveMaster] regionserver.ChunkCreator(498): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-10-17 21:47:18,102 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 42 2018-10-17 21:47:18,534 DEBUG [master/asf904:0:becomeActiveMaster] util.FSUtils(667): Created cluster ID file at hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase.id with ID: f46f5a4b-9b5e-45c4-8281-1ccc0df61f1a 2018-10-17 21:47:18,686 INFO [master/asf904:0:becomeActiveMaster] master.MasterFileSystem(393): BOOTSTRAP: creating hbase:meta region 2018-10-17 21:47:18,708 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(7043): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae Table name == hbase:meta 2018-10-17 21:47:18,951 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 0 2018-10-17 21:47:18,982 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(836): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:19,151 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/info 2018-10-17 21:47:19,216 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:19,280 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:19,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:19,378 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/rep_barrier 2018-10-17 21:47:19,380 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:19,381 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:19,388 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:19,411 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/table 2018-10-17 21:47:19,413 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:19,415 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:19,417 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:19,417 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(949): replaying wal for 1588230740 2018-10-17 21:47:19,455 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740 2018-10-17 21:47:19,459 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/meta/1588230740 2018-10-17 21:47:19,459 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(957): stopping wal replay for 1588230740 2018-10-17 21:47:19,461 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(969): Cleaning up temporary data for 1588230740 2018-10-17 21:47:19,478 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(980): Cleaning up detritus for 1588230740 2018-10-17 21:47:19,489 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-10-17 21:47:19,492 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1005): writing seq id for 1588230740 2018-10-17 21:47:19,534 DEBUG [master/asf904:0:becomeActiveMaster] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:19,534 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1009): Opened 1588230740; next sequenceid=2 2018-10-17 21:47:19,535 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1554): Closing 1588230740, disabling compactions & flushes 2018-10-17 21:47:19,535 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1594): Updates disabled for region hbase:meta,,1.1588230740 2018-10-17 21:47:19,546 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1711): Closed hbase:meta,,1.1588230740 2018-10-17 21:47:19,762 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 0 2018-10-17 21:47:19,780 DEBUG [master/asf904:0:becomeActiveMaster] util.FSTableDescriptors(683): Wrote into hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-10-17 21:47:19,943 INFO [master/asf904:0:becomeActiveMaster] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-17 21:47:20,042 INFO [master/asf904:0:becomeActiveMaster] coordination.ZKSplitLogManagerCoordination(494): Found 0 orphan tasks and 0 rescan nodes 2018-10-17 21:47:20,309 INFO [master/asf904:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x58d320a4 to localhost:63733 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-17 21:47:20,494 DEBUG [master/asf904:0:becomeActiveMaster] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a6fe489, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-17 21:47:20,646 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:20,669 DEBUG [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$DfsBuilderUtility(877): org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder not available, will not use builder API for file creation. 2018-10-17 21:47:20,687 WARN [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(994): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-10-17 21:47:20,688 DEBUG [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(1001): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:990) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:1028) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1078) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:422) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:696) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1505) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:939) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2308) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:582) at java.lang.Thread.run(Thread.java:748) 2018-10-17 21:47:20,706 INFO [master/asf904:0:becomeActiveMaster] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=1 2018-10-17 21:47:20,719 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 69msec 2018-10-17 21:47:20,721 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 1msec 2018-10-17 21:47:20,721 INFO [master/asf904:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(97): Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-10-17 21:47:20,973 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(614): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-10-17 21:47:21,014 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741829_1005{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 39 2018-10-17 21:47:21,050 INFO [master/asf904:0:becomeActiveMaster] master.RegionServerTracker(123): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2018-10-17 21:47:21,197 INFO [master/asf904:0:becomeActiveMaster] balancer.BaseLoadBalancer(1039): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-10-17 21:47:21,220 INFO [master/asf904:0:becomeActiveMaster] balancer.StochasticLoadBalancer(216): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-10-17 21:47:21,318 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-10-17 21:47:21,328 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-10-17 21:47:21,345 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-10-17 21:47:21,354 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-10-17 21:47:21,438 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=2 2018-10-17 21:47:21,458 INFO [WALProcedureStoreSyncThread] wal.ProcedureWALFile(160): Archiving hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000001.log to hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/oldWALs/pv2-00000000000000000001.log 2018-10-17 21:47:21,466 DEBUG [WALProcedureStoreSyncThread] wal.WALProcedureStore(1212): Removed log=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000001.log, activeLogs=[hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log] 2018-10-17 21:47:21,510 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-10-17 21:47:21,510 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-10-17 21:47:21,526 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(797): Active/primary master=asf904.gq1.ygridcore.net,41507,1539812832861, sessionid=0x16683ff2ebf0000, setting cluster-up flag (Was=false) 2018-10-17 21:47:21,694 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-10-17 21:47:21,714 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:21,783 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-10-17 21:47:21,794 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:21,803 WARN [master/asf904:0:becomeActiveMaster] snapshot.SnapshotManager(283): Couldn't delete working snapshot directory: hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.hbase-snapshot/.tmp 2018-10-17 21:47:21,806 INFO [master/asf904:0:becomeActiveMaster] master.ServerManager(1112): No .lastflushedseqids found athdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.lastflushedseqids will record last flushed sequence id for regions by regionserver report all over again 2018-10-17 21:47:21,883 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1012): hbase:meta {1588230740 state=OFFLINE, ts=1539812840977, server=null} 2018-10-17 21:47:21,934 INFO [RS:0;asf904:47370] regionserver.HRegionServer(882): ClusterId : f46f5a4b-9b5e-45c4-8281-1ccc0df61f1a 2018-10-17 21:47:21,968 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-10-17 21:47:21,990 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-10-17 21:47:21,997 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-10-17 21:47:22,055 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-10-17 21:47:22,056 INFO [RS:0;asf904:47370] zookeeper.ReadOnlyZKClient(139): Connect 0x1046d24c to localhost:63733 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-17 21:47:22,200 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1163): Stored pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:22,195 DEBUG [RS:0;asf904:47370] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e812e34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-17 21:47:22,215 DEBUG [RS:0;asf904:47370] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1abeb93c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf904.gq1.ygridcore.net/67.195.81.148:0 2018-10-17 21:47:22,224 DEBUG [RS:0;asf904:47370] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf904:47370 2018-10-17 21:47:22,231 INFO [RS:0;asf904:47370] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-10-17 21:47:22,231 INFO [RS:0;asf904:47370] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-10-17 21:47:22,241 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_OPEN_REGION-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-17 21:47:22,246 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_CLOSE_REGION-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-17 21:47:22,246 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_SERVER_OPERATIONS-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-17 21:47:22,247 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-17 21:47:22,247 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=M_LOG_REPLAY_OPS-master/asf904:0, corePoolSize=10, maxPoolSize=10 2018-10-17 21:47:22,247 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_TABLE_OPERATIONS-master/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-17 21:47:22,249 INFO [RS:0;asf904:47370] regionserver.HRegionServer(2616): reportForDuty to master=asf904.gq1.ygridcore.net,41507,1539812832861 with port=47370, startcode=1539812836748 2018-10-17 21:47:22,263 INFO [master/asf904:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812872263 2018-10-17 21:47:22,272 INFO [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore$DirScanPool(90): Cleaner pool size is 4 2018-10-17 21:47:22,273 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-10-17 21:47:22,275 INFO [master/asf904:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:63733 2018-10-17 21:47:22,318 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-10-17 21:47:22,319 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-10-17 21:47:22,319 INFO [master/asf904:0:becomeActiveMaster] cleaner.LogCleaner(155): Creating OldWALs cleaners with size=2 2018-10-17 21:47:22,357 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-10-17 21:47:22,383 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-10-17 21:47:22,387 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-10-17 21:47:22,398 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.HFileCleaner(225): Starting for large file=Thread[master/asf904:0:becomeActiveMaster-HFileCleaner.large.0-1539812842389,5,FailOnTimeoutGroup] 2018-10-17 21:47:22,398 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner0x0, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-17 21:47:22,406 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.HFileCleaner(240): Starting for small files=Thread[master/asf904:0:becomeActiveMaster-HFileCleaner.small.0-1539812842398,5,FailOnTimeoutGroup] 2018-10-17 21:47:22,414 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(543): replicationLogCleaner-0x16683ff2ebf0004 connected 2018-10-17 21:47:22,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2018-10-17 21:47:22,591 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2018-10-17 21:47:22,664 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-10-17 21:47:22,762 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:57709, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2018-10-17 21:47:22,817 WARN [master/asf904:0] assignment.AssignmentManager(1718): No servers available; cannot place 1 unassigned regions. 2018-10-17 21:47:22,828 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.ServerManager(439): Registering regionserver=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:22,910 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1509): Config from master: hbase.rootdir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:22,914 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1509): Config from master: fs.defaultFS=hdfs://localhost:34726 2018-10-17 21:47:22,914 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1509): Config from master: hbase.master.info.port=-1 2018-10-17 21:47:22,942 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-17 21:47:22,963 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node created, adding [asf904.gq1.ygridcore.net,47370,1539812836748] 2018-10-17 21:47:22,982 DEBUG [RS:0;asf904:47370] zookeeper.ZKUtil(355): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:22,986 WARN [RS:0;asf904:47370] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-10-17 21:47:23,062 DEBUG [RS:0;asf904:47370] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-10-17 21:47:23,070 DEBUG [master/asf904:0] assignment.AssignmentManager(1739): Processing assignQueue; systemServersCount=1, allServersCount=1 2018-10-17 21:47:23,082 INFO [PEWorker-1] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:23,106 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-10-17 21:47:23,119 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:23,317 DEBUG [RS:0;asf904:47370] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:128) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:199) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1817) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1531) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:157) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:961) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) 2018-10-17 21:47:23,319 DEBUG [RS:0;asf904:47370] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-10-17 21:47:23,345 DEBUG [RS:0;asf904:47370] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(196): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:194) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:299) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:129) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:199) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1817) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1531) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:157) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:961) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) 2018-10-17 21:47:23,356 INFO [RS:0;asf904:47370] wal.WALFactory(157): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-10-17 21:47:23,382 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1824): logDir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:23,518 DEBUG [RS:0;asf904:47370] zookeeper.ZKUtil(355): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:23,523 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(746): New admin connection to asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:23,541 DEBUG [RS:0;asf904:47370] regionserver.Replication(144): Replication stats-in-log period=300 seconds 2018-10-17 21:47:23,659 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:41036, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2018-10-17 21:47:23,678 INFO [RS:0;asf904:47370] regionserver.MetricsRegionServerWrapperImpl(147): Computing regionserver metrics every 5000 milliseconds 2018-10-17 21:47:23,804 INFO [RS:0;asf904:47370] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-10-17 21:47:23,842 INFO [RS:0;asf904:47370] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-10-17 21:47:23,852 INFO [RS:0;asf904:47370] regionserver.HRegionServer$CompactionChecker(1710): CompactionChecker runs every PT1S 2018-10-17 21:47:23,852 WARN [RSProcedureDispatcher-pool3-t1] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(179): waiting a little before trying on the same server=asf904.gq1.ygridcore.net,47370,1539812836748 try=0 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:23,899 DEBUG [RSProcedureDispatcher-pool3-t1] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(184): server is not yet up; waiting up to 60000ms org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:23,916 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_OPEN_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-17 21:47:23,919 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_OPEN_META-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-17 21:47:23,919 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-17 21:47:23,919 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_CLOSE_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-17 21:47:23,920 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_CLOSE_META-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-17 21:47:23,920 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf904:0, corePoolSize=2, maxPoolSize=2 2018-10-17 21:47:23,920 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0, corePoolSize=10, maxPoolSize=10 2018-10-17 21:47:23,921 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-17 21:47:23,921 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_REFRESH_PEER-regionserver/asf904:0, corePoolSize=2, maxPoolSize=2 2018-10-17 21:47:23,921 DEBUG [RS:0;asf904:47370] executor.ExecutorService(98): Starting executor service name=RS_REPLAY_SYNC_REPLICATION_WAL-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-17 21:47:24,150 INFO [SplitLogWorker-asf904:47370] regionserver.SplitLogWorker(211): SplitLogWorker asf904.gq1.ygridcore.net,47370,1539812836748 starting 2018-10-17 21:47:24,166 WARN [RSProcedureDispatcher-pool3-t2] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(179): waiting a little before trying on the same server=asf904.gq1.ygridcore.net,47370,1539812836748 try=0 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:24,214 DEBUG [RSProcedureDispatcher-pool3-t2] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(184): server is not yet up; waiting up to 59685ms org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:24,210 INFO [RS:0;asf904:47370] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-10-17 21:47:24,370 WARN [RSProcedureDispatcher-pool3-t3] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(179): waiting a little before trying on the same server=asf904.gq1.ygridcore.net,47370,1539812836748 try=0 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:24,379 DEBUG [RSProcedureDispatcher-pool3-t3] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(184): server is not yet up; waiting up to 59520ms org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:349) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:313) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.ServerNotRunningYetException): org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server asf904.gq1.ygridcore.net,47370,1539812836748 is not running yet at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1509) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3747) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29698) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:24,410 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1550): Serving as asf904.gq1.ygridcore.net,47370,1539812836748, RpcServer on asf904.gq1.ygridcore.net/67.195.81.148:47370, sessionid=0x16683ff2ebf0001 2018-10-17 21:47:24,411 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-10-17 21:47:24,411 DEBUG [RS:0;asf904:47370] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:24,411 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf904.gq1.ygridcore.net,47370,1539812836748' 2018-10-17 21:47:24,411 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-10-17 21:47:24,418 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-10-17 21:47:24,422 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-10-17 21:47:24,422 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-10-17 21:47:24,422 DEBUG [RS:0;asf904:47370] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:24,422 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf904.gq1.ygridcore.net,47370,1539812836748' 2018-10-17 21:47:24,423 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-10-17 21:47:24,438 DEBUG [RS:0;asf904:47370] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-10-17 21:47:24,446 DEBUG [RS:0;asf904:47370] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-10-17 21:47:24,447 INFO [RS:0;asf904:47370] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-10-17 21:47:24,447 INFO [RS:0;asf904:47370] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-10-17 21:47:24,517 INFO [RS_CLOSE_META-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open hbase:meta,,1.1588230740 2018-10-17 21:47:24,526 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.WALFactory(157): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-10-17 21:47:24,596 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf904.gq1.ygridcore.net%2C47370%2C1539812836748.meta, suffix=.meta, logDir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748, archiveDir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/oldWALs 2018-10-17 21:47:24,758 DEBUG [RS-EventLoopGroup-3-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46034,DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef,DISK] 2018-10-17 21:47:24,864 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748/asf904.gq1.ygridcore.net%2C47370%2C1539812836748.meta.1539812844655.meta 2018-10-17 21:47:24,874 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(768): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46034,DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef,DISK]] 2018-10-17 21:47:24,876 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-10-17 21:47:24,976 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-10-17 21:47:25,028 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(8196): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-10-17 21:47:25,061 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-10-17 21:47:25,094 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-10-17 21:47:25,094 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:25,096 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 1588230740 2018-10-17 21:47:25,102 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 1588230740 2018-10-17 21:47:25,158 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/info 2018-10-17 21:47:25,159 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/info 2018-10-17 21:47:25,166 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:25,168 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:25,178 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:25,192 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/rep_barrier 2018-10-17 21:47:25,193 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/rep_barrier 2018-10-17 21:47:25,194 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:25,196 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:25,202 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:25,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/table 2018-10-17 21:47:25,234 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/table 2018-10-17 21:47:25,236 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:25,237 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:25,250 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:25,251 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 1588230740 2018-10-17 21:47:25,278 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740 2018-10-17 21:47:25,326 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/meta/1588230740 2018-10-17 21:47:25,326 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 1588230740 2018-10-17 21:47:25,327 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 1588230740 2018-10-17 21:47:25,342 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 1588230740 2018-10-17 21:47:25,358 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-10-17 21:47:25,374 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 1588230740 2018-10-17 21:47:25,380 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 1588230740; next sequenceid=2 2018-10-17 21:47:25,380 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 1588230740 2018-10-17 21:47:25,583 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionServer(2201): Post open deploy tasks for hbase:meta,,1.1588230740 2018-10-17 21:47:25,587 INFO [RS:0;asf904:47370] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf904.gq1.ygridcore.net%2C47370%2C1539812836748, suffix=, logDir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748, archiveDir=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/oldWALs 2018-10-17 21:47:25,758 DEBUG [RS-EventLoopGroup-3-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46034,DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef,DISK] 2018-10-17 21:47:25,786 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=41507] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:25,866 INFO [RS:0;asf904:47370] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748/asf904.gq1.ygridcore.net%2C47370%2C1539812836748.1539812845602 2018-10-17 21:47:25,874 DEBUG [RS:0;asf904:47370] wal.AbstractFSWAL(768): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46034,DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef,DISK]] 2018-10-17 21:47:25,890 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-10-17 21:47:25,926 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionServer(2225): Finished post open deploy task for hbase:meta,,1.1588230740 2018-10-17 21:47:25,932 INFO [RS_CLOSE_META-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened hbase:meta,,1.1588230740 2018-10-17 21:47:26,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; resume parent processing. 2018-10-17 21:47:26,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=3, ppid=2, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 2.8230sec 2018-10-17 21:47:26,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=1, state=RUNNABLE, hasLock=false; InitMetaProcedure table=hbase:meta; resume parent processing. 2018-10-17 21:47:26,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=2, ppid=1, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 3.6560sec 2018-10-17 21:47:26,243 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta in 4.3100sec 2018-10-17 21:47:26,245 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1048): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1539812837384, completionTime=-1 2018-10-17 21:47:26,245 INFO [master/asf904:0:becomeActiveMaster] master.ServerManager(854): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2018-10-17 21:47:26,245 DEBUG [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:26,517 INFO [RS-EventLoopGroup-3-10] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:41054, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-17 21:47:26,687 INFO [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:26,690 INFO [master/asf904:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812906690 2018-10-17 21:47:26,691 INFO [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1234): Joined the cluster in 445msec 2018-10-17 21:47:27,142 INFO [master/asf904:0:becomeActiveMaster] master.TableNamespaceManager(96): Namespace table not found. Creating... 2018-10-17 21:47:27,162 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(2054): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-10-17 21:47:27,252 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1163): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:27,374 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. from FS 2018-10-17 21:47:27,386 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:27,402 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23 empty. 2018-10-17 21:47:27,410 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:27,410 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'hbase:namespace' archived! 2018-10-17 21:47:27,682 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|FINALIZED]]} size 0 2018-10-17 21:47:27,742 DEBUG [PEWorker-1] util.FSTableDescriptors(683): Wrote into hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-10-17 21:47:27,760 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(7043): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp Table name == hbase:namespace 2018-10-17 21:47:27,971 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 42 2018-10-17 21:47:28,387 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(836): Instantiated hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:28,395 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1554): Closing 031a5042d25e98ddd6a2528d9268ab23, disabling compactions & flushes 2018-10-17 21:47:28,395 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1594): Updates disabled for region hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:28,395 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1711): Closed hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:28,805 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":1539812848530},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1539812848530}]},"ts":1539812848530} 2018-10-17 21:47:29,018 INFO [PEWorker-1] hbase.MetaTableAccessor(1555): Added 1 regions to meta. 2018-10-17 21:47:29,074 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1539812849054}]},"ts":1539812849054} 2018-10-17 21:47:29,106 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2018-10-17 21:47:29,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=031a5042d25e98ddd6a2528d9268ab23, ASSIGN}] 2018-10-17 21:47:29,190 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=031a5042d25e98ddd6a2528d9268ab23, ASSIGN 2018-10-17 21:47:29,198 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=031a5042d25e98ddd6a2528d9268ab23, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47370,1539812836748; forceNewPlan=false, retain=false 2018-10-17 21:47:29,359 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=5 updating hbase:meta row=031a5042d25e98ddd6a2528d9268ab23, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:29,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:29,710 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:29,711 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 031a5042d25e98ddd6a2528d9268ab23, NAME => 'hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23.', STARTKEY => '', ENDKEY => ''} 2018-10-17 21:47:29,713 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,713 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:29,713 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,714 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,826 DEBUG [StoreOpener-031a5042d25e98ddd6a2528d9268ab23-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/info 2018-10-17 21:47:29,827 DEBUG [StoreOpener-031a5042d25e98ddd6a2528d9268ab23-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/info 2018-10-17 21:47:29,829 INFO [StoreOpener-031a5042d25e98ddd6a2528d9268ab23-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:29,831 INFO [StoreOpener-031a5042d25e98ddd6a2528d9268ab23-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:29,846 INFO [StoreOpener-031a5042d25e98ddd6a2528d9268ab23-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:29,848 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,859 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,878 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,878 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,878 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,883 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,910 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:29,994 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:29,994 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 031a5042d25e98ddd6a2528d9268ab23; next sequenceid=2 2018-10-17 21:47:29,995 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 031a5042d25e98ddd6a2528d9268ab23 2018-10-17 21:47:30,026 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2201): Post open deploy tasks for hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:30,080 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=41507] assignment.RegionStateStore(200): pid=5 updating hbase:meta row=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:30,226 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2225): Finished post open deploy task for hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:30,227 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:30,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=031a5042d25e98ddd6a2528d9268ab23, ASSIGN; resume parent processing. 2018-10-17 21:47:30,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=6, ppid=5, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 849msec 2018-10-17 21:47:30,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, hasLock=false; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-10-17 21:47:30,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=5, ppid=4, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=031a5042d25e98ddd6a2528d9268ab23, ASSIGN in 1.1660sec 2018-10-17 21:47:30,403 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1539812850403}]},"ts":1539812850403} 2018-10-17 21:47:30,434 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2018-10-17 21:47:30,514 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-10-17 21:47:30,526 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-10-17 21:47:30,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace in 3.3220sec 2018-10-17 21:47:30,684 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-17 21:47:30,768 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1163): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:31,111 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-10-17 21:47:31,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default in 555msec 2018-10-17 21:47:31,426 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1163): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:31,622 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-10-17 21:47:31,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase in 316msec 2018-10-17 21:47:31,790 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-10-17 21:47:31,826 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-10-17 21:47:31,826 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1111): Master has completed initialization 14.356sec 2018-10-17 21:47:31,835 INFO [master/asf904:0:becomeActiveMaster] quotas.MasterQuotaManager(90): Quota support disabled 2018-10-17 21:47:31,835 INFO [master/asf904:0:becomeActiveMaster] zookeeper.ZKWatcher(205): not a secure deployment, proceeding 2018-10-17 21:47:31,896 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x53d4e507 to localhost:63733 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-17 21:47:31,992 DEBUG [master/asf904:0:becomeActiveMaster] master.HMaster(1169): Balancer post startup initialization complete, took 0 seconds 2018-10-17 21:47:32,030 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@117e0baf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-17 21:47:32,139 INFO [RS-EventLoopGroup-3-11] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:41093, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-17 21:47:32,188 INFO [Time-limited test] hbase.HBaseTestingUtility(1102): Minicluster is up; activeMaster=asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:32,281 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testMergeWithoutPONR Thread=197, OpenFileDescriptor=847, MaxFileDescriptor=60000, SystemLoadAverage=2331, ProcessCount=308, AvailableMemoryMB=5340 2018-10-17 21:47:32,286 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:32,386 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:38404, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2018-10-17 21:47:32,418 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2018-10-17 21:47:32,422 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(422): Client=jenkins//67.195.81.148 set balanceSwitch=false 2018-10-17 21:47:32,428 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:32,463 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.HMaster$4(2018): Client=jenkins//67.195.81.148 create 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-10-17 21:47:32,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] procedure2.ProcedureExecutor(1163): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:32,515 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(630): Client=jenkins//67.195.81.148 procedure request for creating table: namespace: "default" qualifier: "testMergeWithoutPONR" procId is: 9 2018-10-17 21:47:32,525 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. from FS 2018-10-17 21:47:32,528 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:32,534 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2 empty. 2018-10-17 21:47:32,539 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:32,539 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. from FS 2018-10-17 21:47:32,539 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:32,563 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5 empty. 2018-10-17 21:47:32,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:32,582 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:32,582 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. from FS 2018-10-17 21:47:32,583 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:32,587 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a empty. 2018-10-17 21:47:32,598 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:32,598 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. from FS 2018-10-17 21:47:32,599 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae 2018-10-17 21:47:32,602 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f empty. 2018-10-17 21:47:32,605 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:32,606 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'testMergeWithoutPONR' archived! 2018-10-17 21:47:32,670 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|FINALIZED]]} size 0 2018-10-17 21:47:32,685 DEBUG [PEWorker-1] util.FSTableDescriptors(683): Wrote into hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp/data/default/testMergeWithoutPONR/.tabledesc/.tableinfo.0000000001 2018-10-17 21:47:32,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:32,708 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp Table name == testMergeWithoutPONR 2018-10-17 21:47:32,710 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp Table name == testMergeWithoutPONR 2018-10-17 21:47:32,712 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp Table name == testMergeWithoutPONR 2018-10-17 21:47:32,728 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.tmp Table name == testMergeWithoutPONR 2018-10-17 21:47:32,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:32,983 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741837_1013{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 56 2018-10-17 21:47:33,014 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741838_1014{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 56 2018-10-17 21:47:33,015 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741836_1012{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 57 2018-10-17 21:47:33,048 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 0 2018-10-17 21:47:33,051 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:33,052 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1554): Closing 55f7f23154a35661b02f91ff421b58d5, disabling compactions & flushes 2018-10-17 21:47:33,052 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:33,052 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1711): Closed testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:33,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:33,302 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-10-17 21:47:33,303 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-10-17 21:47:33,388 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:33,388 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1554): Closing d630e18e2df7168f95145f7396de1a4f, disabling compactions & flushes 2018-10-17 21:47:33,388 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:33,388 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1711): Closed testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:33,419 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:33,419 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1554): Closing 71fb1fe0ac2b28402e20f8ae99f7339a, disabling compactions & flushes 2018-10-17 21:47:33,419 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:33,419 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1711): Closed testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:33,426 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:33,427 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1554): Closing 3b7371ecf932aa0f7fa0b9a03df56bf2, disabling compactions & flushes 2018-10-17 21:47:33,427 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:33,427 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:33,514 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1539812853506},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1539812853506}]},"ts":1539812853506} 2018-10-17 21:47:33,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":1539812853506},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1539812853506}]},"ts":1539812853506} 2018-10-17 21:47:33,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1539812853506},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1539812853506}]},"ts":1539812853506} 2018-10-17 21:47:33,515 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":1539812853506},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1539812853506}]},"ts":1539812853506} 2018-10-17 21:47:33,695 INFO [PEWorker-1] hbase.MetaTableAccessor(1555): Added 4 regions to meta. 2018-10-17 21:47:33,706 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1539812853706}]},"ts":1539812853706} 2018-10-17 21:47:33,726 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=ENABLING in hbase:meta 2018-10-17 21:47:33,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=71fb1fe0ac2b28402e20f8ae99f7339a, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=d630e18e2df7168f95145f7396de1a4f, ASSIGN}] 2018-10-17 21:47:33,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:33,790 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=d630e18e2df7168f95145f7396de1a4f, ASSIGN 2018-10-17 21:47:33,809 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=d630e18e2df7168f95145f7396de1a4f, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47370,1539812836748; forceNewPlan=false, retain=false 2018-10-17 21:47:33,818 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=71fb1fe0ac2b28402e20f8ae99f7339a, ASSIGN 2018-10-17 21:47:33,834 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=71fb1fe0ac2b28402e20f8ae99f7339a, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47370,1539812836748; forceNewPlan=false, retain=false 2018-10-17 21:47:33,850 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, ASSIGN 2018-10-17 21:47:33,866 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47370,1539812836748; forceNewPlan=false, retain=false 2018-10-17 21:47:33,882 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, ASSIGN 2018-10-17 21:47:33,895 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47370,1539812836748; forceNewPlan=false, retain=false 2018-10-17 21:47:33,981 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=12 updating hbase:meta row=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:33,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:34,043 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=13 updating hbase:meta row=d630e18e2df7168f95145f7396de1a4f, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=15, ppid=13, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:34,135 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=11 updating hbase:meta row=55f7f23154a35661b02f91ff421b58d5, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:34,239 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=10 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,262 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:34,263 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => d630e18e2df7168f95145f7396de1a4f, NAME => 'testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f.', STARTKEY => '2', ENDKEY => ''} 2018-10-17 21:47:34,272 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,272 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:34,272 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,273 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-17 21:47:34,347 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:34,347 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7217): Opening region: {ENCODED => 71fb1fe0ac2b28402e20f8ae99f7339a, NAME => 'testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a.', STARTKEY => '1', ENDKEY => '2'} 2018-10-17 21:47:34,348 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,383 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:34,390 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7256): checking encryption for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,390 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7261): checking classloading for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,398 DEBUG [StoreOpener-d630e18e2df7168f95145f7396de1a4f-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f/FAMILY 2018-10-17 21:47:34,398 DEBUG [StoreOpener-d630e18e2df7168f95145f7396de1a4f-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f/FAMILY 2018-10-17 21:47:34,426 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:34,427 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7217): Opening region: {ENCODED => 55f7f23154a35661b02f91ff421b58d5, NAME => 'testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5.', STARTKEY => '0', ENDKEY => '1'} 2018-10-17 21:47:34,427 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,428 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:34,428 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7256): checking encryption for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,428 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7261): checking classloading for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,435 INFO [StoreOpener-d630e18e2df7168f95145f7396de1a4f-1] hfile.CacheConfig(237): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:34,437 INFO [StoreOpener-d630e18e2df7168f95145f7396de1a4f-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:34,454 INFO [StoreOpener-d630e18e2df7168f95145f7396de1a4f-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:34,474 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,594 DEBUG [StoreOpener-71fb1fe0ac2b28402e20f8ae99f7339a-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a/FAMILY 2018-10-17 21:47:34,594 DEBUG [StoreOpener-71fb1fe0ac2b28402e20f8ae99f7339a-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a/FAMILY 2018-10-17 21:47:34,595 INFO [StoreOpener-71fb1fe0ac2b28402e20f8ae99f7339a-1] hfile.CacheConfig(237): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:34,596 INFO [StoreOpener-71fb1fe0ac2b28402e20f8ae99f7339a-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:34,598 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,610 INFO [StoreOpener-71fb1fe0ac2b28402e20f8ae99f7339a-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:34,614 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(949): replaying wal for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,615 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,615 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,615 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,641 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,662 DEBUG [StoreOpener-55f7f23154a35661b02f91ff421b58d5-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5/FAMILY 2018-10-17 21:47:34,662 DEBUG [StoreOpener-55f7f23154a35661b02f91ff421b58d5-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5/FAMILY 2018-10-17 21:47:34,664 INFO [StoreOpener-55f7f23154a35661b02f91ff421b58d5-1] hfile.CacheConfig(237): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:34,664 INFO [StoreOpener-55f7f23154a35661b02f91ff421b58d5-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:34,671 INFO [StoreOpener-55f7f23154a35661b02f91ff421b58d5-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:34,674 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(949): replaying wal for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,690 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,694 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,702 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,702 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(957): stopping wal replay for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,703 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(969): Cleaning up temporary data for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,703 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,704 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(957): stopping wal replay for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,704 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(969): Cleaning up temporary data for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,710 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(980): Cleaning up detritus for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,726 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,766 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(980): Cleaning up detritus for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,767 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1005): writing seq id for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,786 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:34,802 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened d630e18e2df7168f95145f7396de1a4f; next sequenceid=2 2018-10-17 21:47:34,802 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for d630e18e2df7168f95145f7396de1a4f 2018-10-17 21:47:34,811 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2201): Post open deploy tasks for testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:34,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:34,814 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1005): writing seq id for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,890 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:34,890 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1009): Opened 71fb1fe0ac2b28402e20f8ae99f7339a; next sequenceid=2 2018-10-17 21:47:34,890 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1016): Running coprocessor post-open hooks for 71fb1fe0ac2b28402e20f8ae99f7339a 2018-10-17 21:47:34,898 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] assignment.RegionStateStore(200): pid=13 updating hbase:meta row=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,900 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:34,900 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1009): Opened 55f7f23154a35661b02f91ff421b58d5; next sequenceid=2 2018-10-17 21:47:34,900 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1016): Running coprocessor post-open hooks for 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:34,913 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2201): Post open deploy tasks for testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:34,915 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2201): Post open deploy tasks for testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:34,918 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2225): Finished post open deploy task for testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:34,918 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:34,918 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:34,919 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 3b7371ecf932aa0f7fa0b9a03df56bf2, NAME => 'testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2.', STARTKEY => '', ENDKEY => '0'} 2018-10-17 21:47:34,919 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:34,920 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-17 21:47:34,920 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:34,920 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:34,934 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=41507] assignment.RegionStateStore(200): pid=12 updating hbase:meta row=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=d630e18e2df7168f95145f7396de1a4f, ASSIGN; resume parent processing. 2018-10-17 21:47:34,942 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=15, ppid=13, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 864msec 2018-10-17 21:47:34,951 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] assignment.RegionStateStore(200): pid=11 updating hbase:meta row=55f7f23154a35661b02f91ff421b58d5, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:34,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=13, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=d630e18e2df7168f95145f7396de1a4f, ASSIGN in 1.1950sec 2018-10-17 21:47:34,984 DEBUG [StoreOpener-3b7371ecf932aa0f7fa0b9a03df56bf2-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2/FAMILY 2018-10-17 21:47:34,984 DEBUG [StoreOpener-3b7371ecf932aa0f7fa0b9a03df56bf2-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2/FAMILY 2018-10-17 21:47:34,985 INFO [StoreOpener-3b7371ecf932aa0f7fa0b9a03df56bf2-1] hfile.CacheConfig(237): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-17 21:47:34,998 INFO [StoreOpener-3b7371ecf932aa0f7fa0b9a03df56bf2-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-17 21:47:35,004 INFO [StoreOpener-3b7371ecf932aa0f7fa0b9a03df56bf2-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-10-17 21:47:35,005 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,033 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,034 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2225): Finished post open deploy task for testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:35,034 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:35,034 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2225): Finished post open deploy task for testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:35,035 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:35,042 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,042 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,043 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,044 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,058 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,074 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, ASSIGN; resume parent processing. 2018-10-17 21:47:35,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=16, ppid=11, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 881msec 2018-10-17 21:47:35,083 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-17 21:47:35,083 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 3b7371ecf932aa0f7fa0b9a03df56bf2; next sequenceid=2 2018-10-17 21:47:35,084 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:35,099 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=11, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, ASSIGN in 1.3270sec 2018-10-17 21:47:35,102 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2201): Post open deploy tasks for testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:35,123 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] assignment.RegionStateStore(200): pid=10 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:35,133 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2225): Finished post open deploy task for testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:35,133 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:35,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=71fb1fe0ac2b28402e20f8ae99f7339a, ASSIGN; resume parent processing. 2018-10-17 21:47:35,135 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=14, ppid=12, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 1.1130sec 2018-10-17 21:47:35,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=12, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=71fb1fe0ac2b28402e20f8ae99f7339a, ASSIGN in 1.3870sec 2018-10-17 21:47:35,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, ASSIGN; resume parent processing. 2018-10-17 21:47:35,195 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=17, ppid=10, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 900msec 2018-10-17 21:47:35,231 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR; resume parent processing. 2018-10-17 21:47:35,231 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=10, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, ASSIGN in 1.4470sec 2018-10-17 21:47:35,247 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1539812855246}]},"ts":1539812855246} 2018-10-17 21:47:35,282 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=ENABLED in hbase:meta 2018-10-17 21:47:35,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR in 2.8580sec 2018-10-17 21:47:36,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-17 21:47:36,853 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3722): Operation: CREATE, Table Name: default:testMergeWithoutPONR, procId: 9 completed 2018-10-17 21:47:36,864 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [900,000] milli-secs(wait.for.ratio=[1]) 2018-10-17 21:47:37,210 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: true 2018-10-17 21:47:37,269 DEBUG [Time-limited test] procedure2.ProcedureExecutor(1163): Stored pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,287 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,446 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:37,446 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:37,447 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:37,522 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 3 exec state=pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=true 2018-10-17 21:47:37,524 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:37,525 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:37,606 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741830_1006{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 56914 2018-10-17 21:47:38,026 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:38,044 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:38,054 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:38,075 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:38,108 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 32ms 2018-10-17 21:47:38,109 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:38,174 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=3 2018-10-17 21:47:38,182 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 127msec 2018-10-17 21:47:38,261 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:38,273 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:38,273 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:38,282 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:38,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:38,285 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:38,285 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,295 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:38,296 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,298 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,298 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,300 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 117msec 2018-10-17 21:47:38,330 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812888330 2018-10-17 21:47:38,347 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,348 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:38,363 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:38,364 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:38,364 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:38,365 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:38,366 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:38,366 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:38,367 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812918367 2018-10-17 21:47:38,367 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 19msec 2018-10-17 21:47:38,466 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:38,551 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:38,552 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:38,552 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:38,618 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 4 exec state=pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:38,618 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:38,618 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:38,667 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 470 2018-10-17 21:47:39,089 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:39,102 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:39,110 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:39,142 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:39,157 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 14ms 2018-10-17 21:47:39,157 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:39,203 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:39,212 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 9ms 2018-10-17 21:47:39,212 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:39,258 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=4 2018-10-17 21:47:39,302 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 191msec 2018-10-17 21:47:39,333 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:39,373 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:39,379 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:39,380 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:39,380 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:39,381 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:39,382 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:39,382 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,382 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:39,383 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,383 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,383 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,384 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 80msec 2018-10-17 21:47:39,391 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812889391 2018-10-17 21:47:39,392 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:39,407 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,419 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:39,420 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:39,421 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:39,422 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:39,423 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:39,423 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:39,423 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812919423 2018-10-17 21:47:39,423 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 31msec 2018-10-17 21:47:39,424 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:39,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN}] 2018-10-17 21:47:39,542 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:39,543 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:39,544 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:39,678 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 12 exec state=pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:39,678 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:39,679 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:39,718 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741841_1017{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 474 2018-10-17 21:47:39,780 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-17 21:47:40,122 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:40,130 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:40,138 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:40,148 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:40,154 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 6ms 2018-10-17 21:47:40,154 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:40,186 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:40,194 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 7ms 2018-10-17 21:47:40,195 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:40,210 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:40,219 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log after 7ms 2018-10-17 21:47:40,219 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1539812860110; access_time=1539812859244; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:40,246 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=5 2018-10-17 21:47:40,255 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 116msec 2018-10-17 21:47:40,271 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:40,287 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:40,324 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:40,339 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:40,340 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:40,340 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:40,341 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:40,341 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:40,341 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,350 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:40,351 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,351 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,351 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,352 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 96msec 2018-10-17 21:47:40,382 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812890382 2018-10-17 21:47:40,382 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:40,409 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,431 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:40,432 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:40,433 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:40,433 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:40,439 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:40,439 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:40,439 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812920439 2018-10-17 21:47:40,440 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 57msec 2018-10-17 21:47:40,442 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN}] 2018-10-17 21:47:40,443 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:40,550 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:40,587 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=20 updating hbase:meta row=55f7f23154a35661b02f91ff421b58d5, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:40,618 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-17 21:47:40,619 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:40,619 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:40,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:40,619 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:40,694 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 12 exec state=pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:40,694 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:40,695 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:40,738 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741842_1018{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 1489 2018-10-17 21:47:41,160 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:41,161 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:41,170 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:41,179 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:41,182 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 3ms 2018-10-17 21:47:41,182 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:41,199 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:41,206 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 6ms 2018-10-17 21:47:41,207 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:41,222 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:41,223 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-17 21:47:41,223 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1539812860110; access_time=1539812859244; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:41,230 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:41,232 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-17 21:47:41,232 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=1489; replication=1; blocksize=134217728; modification_time=1539812861150; access_time=1539812860239; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:41,278 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=6 2018-10-17 21:47:41,283 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 112msec 2018-10-17 21:47:41,296 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:41,311 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:41,326 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:41,359 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:41,392 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:41,393 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:41,394 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:41,395 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:41,396 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:41,396 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:41,397 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:41,397 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:41,397 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:41,398 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:41,398 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:41,406 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:41,407 DEBUG [Time-limited test] procedure2.Procedure(965): pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-17 21:47:41,407 DEBUG [Time-limited test] procedure2.Procedure(979): pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:41,407 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:41,408 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 124msec 2018-10-17 21:47:41,409 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2 to restore RIT 2018-10-17 21:47:41,410 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5 to restore RIT 2018-10-17 21:47:41,422 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812891422 2018-10-17 21:47:41,422 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:41,454 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:41,454 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:41,488 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:41,489 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:41,490 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:41,490 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:41,491 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:41,491 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:41,492 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812921492 2018-10-17 21:47:41,492 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 69msec 2018-10-17 21:47:41,496 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=20 updating hbase:meta row=55f7f23154a35661b02f91ff421b58d5, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:41,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-17 21:47:41,518 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:41,710 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:41,730 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:41,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-17 21:47:41,742 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:41,742 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=19, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:41,742 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:41,743 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=19, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:41,767 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 12 exec state=pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:41,767 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:41,767 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:41,870 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(102): Close 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:41,877 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 55f7f23154a35661b02f91ff421b58d5, disabling compactions & flushes 2018-10-17 21:47:41,878 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:41,918 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 688 2018-10-17 21:47:41,926 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:41,927 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:41,934 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:41,947 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:41,954 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 7ms 2018-10-17 21:47:41,955 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:41,973 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/55f7f23154a35661b02f91ff421b58d5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-17 21:47:41,986 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed testMergeWithoutPONR,0,1539812852450.55f7f23154a35661b02f91ff421b58d5. 2018-10-17 21:47:41,990 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:41,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] ipc.CallRunner(142): callId: 23 service: RegionServerStatusService methodName: ReportRegionStateTransition size: 137 connection: 67.195.81.148:57709 deadline: 1539812871991, exception=org.apache.hadoop.hbase.PleaseHoldException: AssignmentManager not running 2018-10-17 21:47:41,995 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 4ms 2018-10-17 21:47:41,995 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:42,006 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2308): Failed report transition server { host_name: "asf904.gq1.ygridcore.net" port: 47370 start_code: 1539812836748 } transition { transition_code: CLOSED region_info { region_id: 1539812852450 table_name { namespace: "default" qualifier: "testMergeWithoutPONR" } start_key: "0" end_key: "1" offline: false split: false replica_id: 0 } }; retry (#0) after 1000ms delay (Master is coming online...). org.apache.hadoop.hbase.PleaseHoldException: org.apache.hadoop.hbase.PleaseHoldException: AssignmentManager not running at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkMetaLoaded(AssignmentManager.java:1307) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:834) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:795) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1582) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15176) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:338) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2299) at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:119) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: AssignmentManager not running at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkMetaLoaded(AssignmentManager.java:1307) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:834) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:795) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1582) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15176) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more 2018-10-17 21:47:42,017 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:42,046 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log after 20ms 2018-10-17 21:47:42,046 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1539812860110; access_time=1539812859244; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:42,067 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:42,082 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log after 14ms 2018-10-17 21:47:42,082 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=1489; replication=1; blocksize=134217728; modification_time=1539812861150; access_time=1539812860239; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:42,115 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:42,126 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log after 11ms 2018-10-17 21:47:42,126 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2079; replication=1; blocksize=134217728; modification_time=1539812861923; access_time=1539812861265; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:42,170 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=7 2018-10-17 21:47:42,183 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 248msec 2018-10-17 21:47:42,202 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:42,219 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:42,239 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:42,247 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:42,269 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:42,281 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:42,286 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:42,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:42,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:42,288 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:42,294 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:42,294 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=21, ppid=20, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-17 21:47:42,295 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:42,295 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:42,296 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:42,301 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:42,302 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:42,302 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:42,302 DEBUG [Time-limited test] procedure2.Procedure(979): pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:42,303 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:42,303 DEBUG [Time-limited test] procedure2.Procedure(979): pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:42,303 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:42,303 DEBUG [Time-limited test] procedure2.Procedure(965): pid=21, ppid=20, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-17 21:47:42,304 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 120msec 2018-10-17 21:47:42,318 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2 to restore RIT 2018-10-17 21:47:42,319 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=20, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5 to restore RIT 2018-10-17 21:47:42,326 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812892326 2018-10-17 21:47:42,326 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:42,342 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:42,379 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:42,380 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:42,380 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:42,381 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:42,381 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:42,382 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:42,450 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812922450 2018-10-17 21:47:42,451 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 124msec 2018-10-17 21:47:42,570 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:42,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1935): Finished subprocedure(s) of pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN; resume parent processing. 2018-10-17 21:47:42,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1546): Finished pid=21, ppid=20, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 1.0520sec 2018-10-17 21:47:42,643 WARN [PEWorker-1] assignment.TransitRegionStateProcedure(326): Failed transition, suspend 1secs pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN; rit=CLOSING, location=asf904.gq1.ygridcore.net,47370,1539812836748; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.HBaseIOException: Failed to close region at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.confirmClosed(TransitRegionStateProcedure.java:256) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:320) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:97) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:284) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:97) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:940) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:42,647 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(82): ADDED pid=20, ppid=18, state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN; timeout=1004, timestamp=1539812863651 2018-10-17 21:47:42,648 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:42,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-17 21:47:42,658 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:42,659 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=19, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:42,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:42,659 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=19, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:42,701 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 12 exec state=pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:42,702 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:42,702 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:42,726 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741844_1020{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 1412 2018-10-17 21:47:43,028 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] assignment.TransitRegionStateProcedure(410): pid=20, ppid=18, state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN received unexpected report transition call from asf904.gq1.ygridcore.net,47370,1539812836748, code=CLOSED, seqId=-1 2018-10-17 21:47:43,030 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2294): TRANSITION REPORTED server { host_name: "asf904.gq1.ygridcore.net" port: 47370 start_code: 1539812836748 } transition { transition_code: CLOSED region_info { region_id: 1539812852450 table_name { namespace: "default" qualifier: "testMergeWithoutPONR" } start_key: "0" end_key: "1" offline: false split: false replica_id: 0 } } 2018-10-17 21:47:43,030 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(124): Closed 55f7f23154a35661b02f91ff421b58d5 2018-10-17 21:47:43,178 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:43,186 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:43,190 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:43,198 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:43,210 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 11ms 2018-10-17 21:47:43,210 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,232 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:43,254 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 22ms 2018-10-17 21:47:43,254 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,268 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:43,274 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log after 5ms 2018-10-17 21:47:43,274 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1539812860110; access_time=1539812859244; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,291 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:43,306 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log after 15ms 2018-10-17 21:47:43,306 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=1489; replication=1; blocksize=134217728; modification_time=1539812861150; access_time=1539812860239; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,327 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:43,334 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log after 7ms 2018-10-17 21:47:43,334 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2079; replication=1; blocksize=134217728; modification_time=1539812861923; access_time=1539812861265; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,359 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log 2018-10-17 21:47:43,374 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log after 15ms 2018-10-17 21:47:43,374 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=1412; replication=1; blocksize=134217728; modification_time=1539812863154; access_time=1539812862162; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:43,431 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=8 2018-10-17 21:47:43,451 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 260msec 2018-10-17 21:47:43,467 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 5 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log 2018-10-17 21:47:43,503 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:43,526 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:43,548 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:43,574 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:43,601 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:43,609 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:43,609 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:43,610 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:43,610 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:43,611 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:43,611 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:43,611 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=21, ppid=20, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-17 21:47:43,611 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:43,612 DEBUG [Time-limited test] procedure2.ProcedureExecutor(561): Loading pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:43,612 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:43,612 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:43,612 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:43,613 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:43,613 DEBUG [Time-limited test] procedure2.Procedure(979): pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-17 21:47:43,613 INFO [Time-limited test] procedure.MasterProcedureScheduler(705): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:43,613 DEBUG [Time-limited test] procedure2.Procedure(965): pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-17 21:47:43,614 DEBUG [Time-limited test] procedure2.Procedure(965): pid=21, ppid=20, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-17 21:47:43,616 INFO [Time-limited test] procedure2.ProcedureExecutor(712): Loaded WALProcedureStore in 164msec 2018-10-17 21:47:43,623 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2 to restore RIT 2018-10-17 21:47:43,624 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN to rit=OFFLINE, location=null, table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5 to restore RIT 2018-10-17 21:47:43,631 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1539812893631 2018-10-17 21:47:43,632 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-17 21:47:43,656 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:43,656 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1536): LOCK_EVENT_WAIT pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:43,675 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=031a5042d25e98ddd6a2528d9268ab23, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:43,676 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:43,676 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=55f7f23154a35661b02f91ff421b58d5, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:43,677 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=71fb1fe0ac2b28402e20f8ae99f7339a, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:43,677 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=d630e18e2df7168f95145f7396de1a4f, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47370,1539812836748, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748, openSeqNum=2 2018-10-17 21:47:43,678 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-17 21:47:43,687 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1539812923687 2018-10-17 21:47:43,687 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 55msec 2018-10-17 21:47:43,682 INFO [PEWorker-1] procedure.MasterProcedureScheduler(705): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:43,838 WARN [PEWorker-1] assignment.TransitRegionStateProcedure(326): Failed transition, suspend 1secs pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN; rit=CLOSING, location=asf904.gq1.ygridcore.net,47370,1539812836748; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.HBaseIOException: Failed to close region at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.confirmClosed(TransitRegionStateProcedure.java:256) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:320) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:97) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:284) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:97) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:940) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1760) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:43,839 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(82): ADDED pid=20, ppid=18, state=WAITING_TIMEOUT:REGION_STATE_TRANSITION_CLOSE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN; timeout=1006, timestamp=1539812864845 2018-10-17 21:47:43,839 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:43,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1795): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-17 21:47:43,883 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: true 2018-10-17 21:47:44,126 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(102): Close 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:44,166 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1554): Closing 3b7371ecf932aa0f7fa0b9a03df56bf2, disabling compactions & flushes 2018-10-17 21:47:44,178 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:44,258 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/3b7371ecf932aa0f7fa0b9a03df56bf2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-17 21:47:44,278 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,,1539812852450.3b7371ecf932aa0f7fa0b9a03df56bf2. 2018-10-17 21:47:44,280 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=41507] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=3b7371ecf932aa0f7fa0b9a03df56bf2, regionState=CLOSED 2018-10-17 21:47:44,314 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(124): Closed 3b7371ecf932aa0f7fa0b9a03df56bf2 2018-10-17 21:47:44,338 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(123): Toggle KILL before store update to: false 2018-10-17 21:47:44,338 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1870): TESTING: Kill BEFORE store update: pid=22, ppid=19, state=SUCCESS, hasLock=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-17 21:47:44,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(743): Stopping 2018-10-17 21:47:44,339 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2082): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=22, ppid=19, state=SUCCESS, hasLock=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1872) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1816) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1528) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:80) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2066) 2018-10-17 21:47:44,446 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(407): Restart 12 exec state=pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:44,446 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-17 21:47:44,446 INFO [Time-limited test] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:44,470 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|FINALIZED]]} size 359 2018-10-17 21:47:44,478 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-17 21:47:44,486 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-17 21:47:44,488 INFO [Time-limited test] procedure2.ProcedureExecutor(679): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-17 21:47:44,499 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:44,506 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log after 7ms 2018-10-17 21:47:44,507 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=56914; replication=1; blocksize=134217728; modification_time=1539812858014; access_time=1539812840734; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,538 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:44,582 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log after 43ms 2018-10-17 21:47:44,583 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1539812859082; access_time=1539812858161; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,604 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:44,610 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log after 6ms 2018-10-17 21:47:44,610 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1539812860110; access_time=1539812859244; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,620 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:44,626 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log after 5ms 2018-10-17 21:47:44,626 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=1489; replication=1; blocksize=134217728; modification_time=1539812861150; access_time=1539812860239; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,671 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:44,679 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log after 8ms 2018-10-17 21:47:44,679 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2079; replication=1; blocksize=134217728; modification_time=1539812861923; access_time=1539812861265; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,740 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log 2018-10-17 21:47:44,754 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log after 13ms 2018-10-17 21:47:44,754 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=1412; replication=1; blocksize=134217728; modification_time=1539812863154; access_time=1539812862162; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,774 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000008.log 2018-10-17 21:47:44,786 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000008.log after 11ms 2018-10-17 21:47:44,786 DEBUG [Time-limited test] wal.WALProcedureStore(1343): Opening Pv2 FileStatus{path=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2420; replication=1; blocksize=134217728; modification_time=1539812864474; access_time=1539812863418; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-17 21:47:44,838 INFO [Time-limited test] wal.WALProcedureStore(1123): Rolled new Procedure Store WAL, id=9 2018-10-17 21:47:44,847 INFO [Time-limited test] procedure2.ProcedureExecutor(698): Recovered WALProcedureStore lease in 358msec 2018-10-17 21:47:44,857 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000008.log 2018-10-17 21:47:44,873 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 5 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000007.log 2018-10-17 21:47:44,892 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000006.log 2018-10-17 21:47:44,915 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000005.log 2018-10-17 21:47:44,946 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000004.log 2018-10-17 21:47:44,966 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000003.log 2018-10-17 21:47:44,986 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/MasterProcWALs/pv2-00000000000000000002.log 2018-10-17 21:47:44,996 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-17 21:47:44,997 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-17 21:47:44,997 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-17 21:47:44,997 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-17 21:47:44,998 DEBUG [Time-limited test] procedure2.ProcedureExecutor(512): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-17 21:47:44,999 ERROR [Time-limited test] procedure2.ProcedureExecutor$2(444): Corrupt pid=22, ppid=19, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-17 21:47:45,007 ERROR [Time-limited test] procedure2.ProcedureExecutor$2(444): Corrupt pid=19, ppid=18, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=3b7371ecf932aa0f7fa0b9a03df56bf2, UNASSIGN 2018-10-17 21:47:45,008 ERROR [Time-limited test] procedure2.ProcedureExecutor$2(444): Corrupt pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=55f7f23154a35661b02f91ff421b58d5, UNASSIGN 2018-10-17 21:47:45,009 ERROR [Time-limited test] procedure2.ProcedureExecutor$2(444): Corrupt pid=21, ppid=20, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-17 21:47:45,010 ERROR [Time-limited test] procedure2.ProcedureExecutor$2(444): Corrupt pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[3b7371ecf932aa0f7fa0b9a03df56bf2, 55f7f23154a35661b02f91ff421b58d5], forcibly=false 2018-10-17 21:47:45,022 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,158 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testMergeWithoutPONR Thread=202 (was 197) Potentially hanging thread: hconnection-0x2509c7b9-shared-pool6-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf904:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_777192731_23 at /127.0.0.1:59735 [Waiting for operation #113] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf904:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x2509c7b9-shared-pool6-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Thread-186 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:523) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=842 (was 847), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=2435 (was 2331) - SystemLoadAverage LEAK? -, ProcessCount=311 (was 308) - ProcessCount LEAK? -, AvailableMemoryMB=5209 (was 5340) 2018-10-17 21:47:45,238 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testRollbackAndDoubleExecution Thread=202, OpenFileDescriptor=844, MaxFileDescriptor=60000, SystemLoadAverage=2435, ProcessCount=308, AvailableMemoryMB=5204 2018-10-17 21:47:45,239 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,239 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,325 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testRollbackAndDoubleExecution Thread=202 (was 202), OpenFileDescriptor=842 (was 844), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=2435 (was 2435), ProcessCount=309 (was 308) - ProcessCount LEAK? -, AvailableMemoryMB=5197 (was 5204) 2018-10-17 21:47:45,410 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testMergeTwoRegions Thread=202, OpenFileDescriptor=842, MaxFileDescriptor=60000, SystemLoadAverage=2435, ProcessCount=309, AvailableMemoryMB=5191 2018-10-17 21:47:45,411 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,411 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,494 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testMergeTwoRegions Thread=201 (was 202), OpenFileDescriptor=842 (was 842), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=2435 (was 2435), ProcessCount=309 (was 309), AvailableMemoryMB=5184 (was 5191) 2018-10-17 21:47:45,578 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testMergeRegionsConcurrently Thread=201, OpenFileDescriptor=842, MaxFileDescriptor=60000, SystemLoadAverage=2435, ProcessCount=309, AvailableMemoryMB=5172 2018-10-17 21:47:45,578 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,579 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,666 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testMergeRegionsConcurrently Thread=201 (was 201), OpenFileDescriptor=842 (was 842), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=2435 (was 2435), ProcessCount=310 (was 309) - ProcessCount LEAK? -, AvailableMemoryMB=5161 (was 5172) 2018-10-17 21:47:45,746 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testRecoveryAndDoubleExecution Thread=201, OpenFileDescriptor=842, MaxFileDescriptor=60000, SystemLoadAverage=2435, ProcessCount=309, AvailableMemoryMB=5153 2018-10-17 21:47:45,747 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,747 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-17 21:47:45,795 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testRecoveryAndDoubleExecution Thread=201 (was 201), OpenFileDescriptor=845 (was 842) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=2435 (was 2435), ProcessCount=309 (was 309), AvailableMemoryMB=5148 (was 5153) 2018-10-17 21:47:45,795 INFO [Time-limited test] hbase.HBaseTestingUtility(1228): Shutting down minicluster 2018-10-17 21:47:45,796 INFO [Time-limited test] client.ConnectionImplementation(1801): Closing master protocol: MasterService 2018-10-17 21:47:45,796 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x53d4e507 to localhost:63733 2018-10-17 21:47:45,796 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-17 21:47:45,798 DEBUG [Time-limited test] util.JVMClusterUtil(247): Shutting down HBase Cluster 2018-10-17 21:47:45,804 INFO [Time-limited test] master.ServerManager(916): Cluster shutdown requested of master=asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:45,830 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-10-17 21:47:45,842 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x58d320a4 to localhost:63733 2018-10-17 21:47:45,850 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-17 21:47:45,851 INFO [Time-limited test] regionserver.HRegionServer(2163): ***** STOPPING region server 'asf904.gq1.ygridcore.net,47370,1539812836748' ***** 2018-10-17 21:47:45,851 INFO [Time-limited test] regionserver.HRegionServer(2177): STOPPED: Shutdown requested 2018-10-17 21:47:45,851 INFO [RS:0;asf904:47370] regionserver.SplitLogWorker(241): Sending interrupt to stop the worker thread 2018-10-17 21:47:45,852 INFO [RS:0;asf904:47370] regionserver.HeapMemoryManager(221): Stopping 2018-10-17 21:47:45,853 INFO [RS:0;asf904:47370] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-10-17 21:47:45,853 INFO [RS:0;asf904:47370] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-10-17 21:47:45,830 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-10-17 21:47:45,950 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 031a5042d25e98ddd6a2528d9268ab23, disabling compactions & flushes 2018-10-17 21:47:45,958 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:45,959 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(2647): Flushing 1/1 column families, dataSize=78 B heapSize=488 B 2018-10-17 21:47:45,906 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1073): stopping server asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:45,852 INFO [SplitLogWorker-asf904:47370] regionserver.SplitLogWorker(223): SplitLogWorker interrupted. Exiting. 2018-10-17 21:47:45,963 INFO [SplitLogWorker-asf904:47370] regionserver.SplitLogWorker(232): SplitLogWorker asf904.gq1.ygridcore.net,47370,1539812836748 exiting 2018-10-17 21:47:45,895 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-10-17 21:47:45,851 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-17 21:47:45,963 DEBUG [RS:0;asf904:47370] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-10-17 21:47:45,994 INFO [RS:0;asf904:47370] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x1046d24c to localhost:63733 2018-10-17 21:47:45,994 DEBUG [RS:0;asf904:47370] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-17 21:47:45,995 INFO [RS:0;asf904:47370] regionserver.CompactSplit(431): Waiting for Split Thread to finish... 2018-10-17 21:47:45,995 INFO [RS:0;asf904:47370] regionserver.CompactSplit(431): Waiting for Large Compaction Thread to finish... 2018-10-17 21:47:45,995 INFO [RS:0;asf904:47370] regionserver.CompactSplit(431): Waiting for Small Compaction Thread to finish... 2018-10-17 21:47:45,995 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1403): Waiting on 4 regions to close 2018-10-17 21:47:45,995 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1407): Online Regions={71fb1fe0ac2b28402e20f8ae99f7339a=testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a., 1588230740=hbase:meta,,1.1588230740, 031a5042d25e98ddd6a2528d9268ab23=hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23., d630e18e2df7168f95145f7396de1a4f=testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f.} 2018-10-17 21:47:45,962 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-17 21:47:45,974 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-10-17 21:47:45,998 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1554): Closing d630e18e2df7168f95145f7396de1a4f, disabling compactions & flushes 2018-10-17 21:47:46,006 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:46,006 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 1588230740, disabling compactions & flushes 2018-10-17 21:47:46,007 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region hbase:meta,,1.1588230740 2018-10-17 21:47:46,009 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1554): Closing 71fb1fe0ac2b28402e20f8ae99f7339a, disabling compactions & flushes 2018-10-17 21:47:46,014 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:46,018 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(2647): Flushing 3/3 column families, dataSize=8.64 KB heapSize=14.62 KB 2018-10-17 21:47:46,022 INFO [regionserver/asf904:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-10-17 21:47:46,022 INFO [regionserver/asf904:0.Chore.1] hbase.ScheduledChore(180): Chore: CompactionChecker was stopped 2018-10-17 21:47:46,074 INFO [regionserver/asf904:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-10-17 21:47:46,250 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/d630e18e2df7168f95145f7396de1a4f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-17 21:47:46,258 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/default/testMergeWithoutPONR/71fb1fe0ac2b28402e20f8ae99f7339a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-17 21:47:46,323 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1711): Closed testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:46,330 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.CloseRegionHandler(129): Closed testMergeWithoutPONR,1,1539812852450.71fb1fe0ac2b28402e20f8ae99f7339a. 2018-10-17 21:47:46,331 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:46,331 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.CloseRegionHandler(129): Closed testMergeWithoutPONR,2,1539812852450.d630e18e2df7168f95145f7396de1a4f. 2018-10-17 21:47:46,526 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741847_1023{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 11751 2018-10-17 21:47:46,538 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741846_1022{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 4896 2018-10-17 21:47:46,939 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=8.45 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/.tmp/info/d938ace075b641ccab3ee0da3749b86f 2018-10-17 21:47:46,944 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/.tmp/info/f0d0e75636994e418f94ea5a1d13583d 2018-10-17 21:47:47,022 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1403): Waiting on 2 regions to close 2018-10-17 21:47:47,022 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1407): Online Regions={1588230740=hbase:meta,,1.1588230740, 031a5042d25e98ddd6a2528d9268ab23=hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23.} 2018-10-17 21:47:47,362 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/.tmp/info/f0d0e75636994e418f94ea5a1d13583d as hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/info/f0d0e75636994e418f94ea5a1d13583d 2018-10-17 21:47:47,485 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-17 21:47:47,498 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741848_1024{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 4861 2018-10-17 21:47:47,652 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HStore(1072): Added hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/info/f0d0e75636994e418f94ea5a1d13583d, entries=2, sequenceid=6, filesize=4.8 K 2018-10-17 21:47:47,719 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(2856): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 031a5042d25e98ddd6a2528d9268ab23 in 1760ms, sequenceid=6, compaction requested=false 2018-10-17 21:47:47,722 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.MetricsTableSourceImpl(124): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2018-10-17 21:47:47,882 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/namespace/031a5042d25e98ddd6a2528d9268ab23/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2018-10-17 21:47:47,910 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:47,910 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.CloseRegionHandler(129): Closed hbase:namespace,,1539812847144.031a5042d25e98ddd6a2528d9268ab23. 2018-10-17 21:47:47,912 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=198 B at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/.tmp/table/b4443999b03644b787b68407b9365181 2018-10-17 21:47:48,010 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/.tmp/info/d938ace075b641ccab3ee0da3749b86f as hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/info/d938ace075b641ccab3ee0da3749b86f 2018-10-17 21:47:48,050 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HStore(1072): Added hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/info/d938ace075b641ccab3ee0da3749b86f, entries=54, sequenceid=26, filesize=11.5 K 2018-10-17 21:47:48,054 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1403): Waiting on 1 regions to close 2018-10-17 21:47:48,055 DEBUG [RS:0;asf904:47370] regionserver.HRegionServer(1407): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-10-17 21:47:48,065 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/.tmp/table/b4443999b03644b787b68407b9365181 as hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/table/b4443999b03644b787b68407b9365181 2018-10-17 21:47:48,139 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HStore(1072): Added hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/data/hbase/meta/1588230740/table/b4443999b03644b787b68407b9365181, entries=4, sequenceid=26, filesize=4.7 K 2018-10-17 21:47:48,144 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(2856): Finished flush of dataSize ~8.64 KB/8852, heapSize ~14.34 KB/14680, currentSize=0 B/0 for 1588230740 in 2134ms, sequenceid=26, compaction requested=false 2018-10-17 21:47:48,144 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.MetricsTableSourceImpl(124): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2018-10-17 21:47:48,230 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=1 2018-10-17 21:47:48,238 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-10-17 21:47:48,240 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed hbase:meta,,1.1588230740 2018-10-17 21:47:48,241 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] handler.CloseRegionHandler(129): Closed hbase:meta,,1.1588230740 2018-10-17 21:47:48,258 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1101): stopping server asf904.gq1.ygridcore.net,47370,1539812836748; all regions closed. 2018-10-17 21:47:48,314 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 0 2018-10-17 21:47:48,355 DEBUG [RS:0;asf904:47370] wal.AbstractFSWAL(853): Moved 1 WAL file(s) to /user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/oldWALs 2018-10-17 21:47:48,356 INFO [RS:0;asf904:47370] wal.AbstractFSWAL(856): Closed WAL: AsyncFSWAL asf904.gq1.ygridcore.net%2C47370%2C1539812836748.meta:.meta(num 1539812844655) 2018-10-17 21:47:48,390 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 3895 2018-10-17 21:47:48,394 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/WALs/asf904.gq1.ygridcore.net,47370,1539812836748/asf904.gq1.ygridcore.net%2C47370%2C1539812836748.1539812845602 not finished, retry = 0 2018-10-17 21:47:48,528 DEBUG [RS:0;asf904:47370] wal.AbstractFSWAL(853): Moved 1 WAL file(s) to /user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/oldWALs 2018-10-17 21:47:48,528 INFO [RS:0;asf904:47370] wal.AbstractFSWAL(856): Closed WAL: AsyncFSWAL asf904.gq1.ygridcore.net%2C47370%2C1539812836748:(num 1539812845602) 2018-10-17 21:47:48,529 DEBUG [RS:0;asf904:47370] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-17 21:47:48,530 INFO [RS:0;asf904:47370] regionserver.Leases(149): Closed leases 2018-10-17 21:47:48,535 INFO [RS:0;asf904:47370] hbase.ChoreService(327): Chore service for: regionserver/asf904:0 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-10-17 21:47:48,537 INFO [regionserver/asf904:0.logRoller] regionserver.LogRoller(222): LogRoller exiting. 2018-10-17 21:47:48,560 INFO [RS:0;asf904:47370] ipc.NettyRpcServer(144): Stopping server on /67.195.81.148:47370 2018-10-17 21:47:48,637 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf904.gq1.ygridcore.net,47370,1539812836748 2018-10-17 21:47:48,637 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47370-0x16683ff2ebf0001, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-17 21:47:48,637 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-17 21:47:48,654 INFO [RegionServerTracker-0] master.RegionServerTracker(168): RegionServer ephemeral node deleted, processing expiration [asf904.gq1.ygridcore.net,47370,1539812836748] 2018-10-17 21:47:48,655 DEBUG [RegionServerTracker-0] master.DeadServer(134): Added asf904.gq1.ygridcore.net,47370,1539812836748; numProcessing=1 2018-10-17 21:47:48,655 INFO [RegionServerTracker-0] master.ServerManager(597): Cluster shutdown set; asf904.gq1.ygridcore.net,47370,1539812836748 expired; onlineServers=0 2018-10-17 21:47:48,655 INFO [RegionServerTracker-0] regionserver.HRegionServer(2163): ***** STOPPING region server 'asf904.gq1.ygridcore.net,41507,1539812832861' ***** 2018-10-17 21:47:48,655 INFO [RegionServerTracker-0] regionserver.HRegionServer(2177): STOPPED: Cluster shutdown set; onlineServer=0 2018-10-17 21:47:48,656 INFO [RS:0;asf904:47370] regionserver.HRegionServer(1157): Exiting; stopping=asf904.gq1.ygridcore.net,47370,1539812836748; zookeeper connection closed. 2018-10-17 21:47:48,656 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@49787d7f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(222): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@49787d7f 2018-10-17 21:47:48,658 INFO [Time-limited test] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2018-10-17 21:47:48,665 DEBUG [M:0;asf904:41507] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bf5ab7e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf904.gq1.ygridcore.net/67.195.81.148:0 2018-10-17 21:47:48,665 INFO [M:0;asf904:41507] regionserver.HRegionServer(1073): stopping server asf904.gq1.ygridcore.net,41507,1539812832861 2018-10-17 21:47:48,665 DEBUG [M:0;asf904:41507] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-10-17 21:47:48,665 INFO [M:0;asf904:41507] regionserver.HRegionServer(1101): stopping server asf904.gq1.ygridcore.net,41507,1539812832861; all regions closed. 2018-10-17 21:47:48,665 DEBUG [M:0;asf904:41507] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-17 21:47:48,667 INFO [M:0;asf904:41507] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-10-17 21:47:48,667 INFO [M:0;asf904:41507] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-10-17 21:47:48,667 INFO [M:0;asf904:41507] hbase.ChoreService(327): Chore service for: master/asf904:0 had [[ScheduledChore: Name: FlushedSequenceIdFlusher Period: 10800000 Unit: MILLISECONDS]] on shutdown 2018-10-17 21:47:48,668 DEBUG [M:0;asf904:41507] master.HMaster(1448): Stopping service threads 2018-10-17 21:47:48,690 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-10-17 21:47:48,698 DEBUG [M:0;asf904:41507] zookeeper.RecoverableZooKeeper(176): Node /hbase/master already deleted, retry=false 2018-10-17 21:47:48,699 DEBUG [M:0;asf904:41507] master.ActiveMasterManager(280): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Failed delete of our master address node; KeeperErrorCode = NoNode for /hbase/master 2018-10-17 21:47:48,702 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:41507-0x16683ff2ebf0000, quorum=localhost:63733, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-10-17 21:47:48,710 INFO [M:0;asf904:41507] master.ServerManager(1064): Writing .lastflushedseqids file at: hdfs://localhost:34726/user/jenkins/test-data/111709fe-9132-af84-d0c4-a318bf69e4ae/.lastflushedseqids 2018-10-17 21:47:48,866 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741849_1025{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dff300b3-7c76-4304-abbb-5c2f1d79d4ef:NORMAL:127.0.0.1:46034|RBW]]} size 267 2018-10-17 21:47:49,043 INFO [master/asf904:0.splitLogManager..Chore.1] hbase.ScheduledChore(180): Chore: SplitLogManager Timeout Monitor was stopped 2018-10-17 21:47:49,278 INFO [M:0;asf904:41507] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-10-17 21:47:49,318 INFO [M:0;asf904:41507] wal.WALProcedureStore(330): Stopping the WAL Procedure Store, isAbort=false 2018-10-17 21:47:49,446 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46034 is added to blk_1073741850_1026{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cfcf6b71-0581-475e-be94-80a5198ae438:NORMAL:127.0.0.1:46034|RBW]]} size 47 2018-10-17 21:47:49,870 INFO [M:0;asf904:41507] hbase.ChoreService(327): Chore service for: master/asf904:0.splitLogManager. had [] on shutdown 2018-10-17 21:47:49,870 INFO [M:0;asf904:41507] flush.MasterFlushTableProcedureManager(81): stop: server shutting down. 2018-10-17 21:47:49,877 INFO [M:0;asf904:41507] ipc.NettyRpcServer(144): Stopping server on /67.195.81.148:41507 2018-10-17 21:47:49,918 DEBUG [M:0;asf904:41507] zookeeper.RecoverableZooKeeper(176): Node /hbase/rs/asf904.gq1.ygridcore.net,41507,1539812832861 already deleted, retry=false 2018-10-17 21:47:49,946 INFO [M:0;asf904:41507] regionserver.HRegionServer(1157): Exiting; stopping=asf904.gq1.ygridcore.net,41507,1539812832861; zookeeper connection closed. 2018-10-17 21:47:49,958 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-10-17 21:47:50,424 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-10-17 21:47:50,486 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:34726] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-10-17 21:47:50,510 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/249d8749-bb98-4e65-a4e4-e05df8fdbc1d/cluster_df699823-e035-75e6-0bfa-39b6f75f6b9b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:34726] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-762280921-67.195.81.148-1539812816771 (Datanode Uuid ea0c7941-9642-499f-935e-afb50ff6339a) service to localhost/127.0.0.1:34726 2018-10-17 21:47:50,808 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-10-17 21:47:50,958 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-10-17 21:47:50,975 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(326): Shutdown MiniZK cluster with all ZK servers 2018-10-17 21:47:51,028 INFO [Time-limited test] hbase.HBaseTestingUtility(1235): Minicluster is down 2018-10-17 21:47:51,046 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner-0x16683ff2ebf0004, quorum=localhost:63733, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-10-17 21:47:51,054 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(548): replicationLogCleaner-0x16683ff2ebf0004, quorum=localhost:63733, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-10-17 21:47:51,286 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@2d44f6b8 2018-10-17 21:47:51,298 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-10-17 21:47:51,342 WARN [Thread-98] hdfs.DFSOutputStream$DataStreamer(663): DataStreamer Exception java.io.EOFException: End of File Exception between local host is: "asf904.gq1.ygridcore.net/67.195.81.148"; destination host is: "localhost":34726; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy30.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy31.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy34.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy34.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1603) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1388) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:554) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1085) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:980) 2018-10-17 21:47:51,351 ERROR [ClientFinalizer-shutdown-hook] hdfs.DFSClient(949): Failed to close inode 16419 java.io.EOFException: End of File Exception between local host is: "asf904.gq1.ygridcore.net/67.195.81.148"; destination host is: "localhost":34726; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy30.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy31.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy34.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy34.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1603) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1388) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:554) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1085) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:980) 2018-10-17 21:47:51,354 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.