2018-10-08 03:44:03,099 DEBUG [main] hbase.HBaseTestingUtility(351): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d 2018-10-08 03:44:03,112 INFO [Time-limited test] hbase.HBaseTestingUtility(1031): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2018-10-08 03:44:03,113 INFO [Time-limited test] hbase.HBaseZKTestingUtility(85): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0, deleteOnExit=true 2018-10-08 03:44:03,113 INFO [Time-limited test] hbase.HBaseTestingUtility(1045): STARTING DFS 2018-10-08 03:44:03,114 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cache_data in system properties and HBase conf 2018-10-08 03:44:03,115 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/hadoop_tmp in system properties and HBase conf 2018-10-08 03:44:03,115 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/hadoop_logs in system properties and HBase conf 2018-10-08 03:44:03,116 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/mapred_local in system properties and HBase conf 2018-10-08 03:44:03,117 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/mapred_temp in system properties and HBase conf 2018-10-08 03:44:03,117 INFO [Time-limited test] hbase.HBaseTestingUtility(744): read short circuit is OFF 2018-10-08 03:44:03,224 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-10-08 03:44:03,689 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-10-08 03:44:05,215 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-10-08 03:44:05,413 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-10-08 03:44:05,505 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-10-08 03:44:05,547 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/local-repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_43017_hdfs____.jh1hwf/webapp 2018-10-08 03:44:05,770 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43017 2018-10-08 03:44:06,467 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-10-08 03:44:06,477 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/local-repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_34341_datanode____.6ij6wa/webapp 2018-10-08 03:44:06,635 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34341 2018-10-08 03:44:08,089 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:54583] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-10-08 03:44:08,233 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1be2411bed27d0: from storage DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3 node DatanodeRegistration(127.0.0.1:41173, datanodeUuid=199d283d-ce42-4c67-a4e7-057828b1ea76, infoPort=48535, infoSecurePort=0, ipcPort=44213, storageInfo=lv=-56;cid=testClusterID;nsid=218737283;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs 2018-10-08 03:44:08,234 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1be2411bed27d0: from storage DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a node DatanodeRegistration(127.0.0.1:41173, datanodeUuid=199d283d-ce42-4c67-a4e7-057828b1ea76, infoPort=48535, infoSecurePort=0, ipcPort=44213, storageInfo=lv=-56;cid=testClusterID;nsid=218737283;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-10-08 03:44:08,416 DEBUG [Time-limited test] hbase.HBaseTestingUtility(679): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d 2018-10-08 03:44:08,491 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-10-08 03:44:08,511 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran successful 'stat' on client port=63142 2018-10-08 03:44:08,529 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:08,533 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:08,907 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 7 2018-10-08 03:44:09,318 INFO [Time-limited test] util.FSUtils(515): Created version file at hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 with version=8 2018-10-08 03:44:09,318 INFO [Time-limited test] hbase.HBaseTestingUtility(1345): Setting hbase.fs.tmp.dir to hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase-staging 2018-10-08 03:44:09,592 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-10-08 03:44:09,887 INFO [Time-limited test] client.ConnectionUtils(122): master/asf904:0 server-side Connection retries=45 2018-10-08 03:44:09,916 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-10-08 03:44:09,918 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-10-08 03:44:09,918 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-10-08 03:44:10,065 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientService, hbase.pb.AdminService 2018-10-08 03:44:10,253 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-10-08 03:44:10,342 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /67.195.81.148:38583 2018-10-08 03:44:10,356 INFO [Time-limited test] hfile.CacheConfig(553): Allocating onheap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-10-08 03:44:10,362 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:10,364 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:10,367 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-10-08 03:44:10,368 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:10,372 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:10,685 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:38583 connecting to ZooKeeper ensemble=localhost:63142 2018-10-08 03:44:10,728 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:385830x0, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-08 03:44:10,729 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): master:38583-0x16651c665170000 connected 2018-10-08 03:44:10,770 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-10-08 03:44:10,772 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-08 03:44:10,783 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38583 2018-10-08 03:44:10,784 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=38583 2018-10-08 03:44:10,785 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38583 2018-10-08 03:44:10,795 INFO [Time-limited test] master.HMaster(504): hbase.rootdir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904, hbase.cluster.distributed=false 2018-10-08 03:44:10,892 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf904:0 server-side Connection retries=45 2018-10-08 03:44:10,893 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-10-08 03:44:10,893 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-10-08 03:44:10,893 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-10-08 03:44:10,898 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-10-08 03:44:10,899 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-10-08 03:44:10,902 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /67.195.81.148:47093 2018-10-08 03:44:10,903 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:10,904 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:10,907 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:10,911 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:10,916 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:47093 connecting to ZooKeeper ensemble=localhost:63142 2018-10-08 03:44:10,930 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:470930x0, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-08 03:44:10,930 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:47093-0x16651c665170001 connected 2018-10-08 03:44:10,931 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-10-08 03:44:10,932 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-10-08 03:44:10,933 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=47093 2018-10-08 03:44:10,935 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=47093 2018-10-08 03:44:10,936 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=47093 2018-10-08 03:44:10,940 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(2274): Adding backup master ZNode /hbase/backup-masters/asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:10,951 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:10,987 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-10-08 03:44:10,987 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-10-08 03:44:10,989 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-10-08 03:44:10,989 INFO [master/asf904:0:becomeActiveMaster] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf904.gq1.ygridcore.net,38583,1538970249417 from backup master directory 2018-10-08 03:44:10,993 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-10-08 03:44:10,993 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:10,994 WARN [master/asf904:0:becomeActiveMaster] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-10-08 03:44:10,994 INFO [master/asf904:0:becomeActiveMaster] master.ActiveMasterManager(181): Registered as active master=asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:10,998 INFO [master/asf904:0:becomeActiveMaster] regionserver.ChunkCreator(498): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-10-08 03:44:10,999 INFO [master/asf904:0:becomeActiveMaster] regionserver.ChunkCreator(498): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-10-08 03:44:11,170 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 42 2018-10-08 03:44:11,573 DEBUG [master/asf904:0:becomeActiveMaster] util.FSUtils(667): Created cluster ID file at hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase.id with ID: 312144ec-6fe1-49a3-857f-b64e3d5476c9 2018-10-08 03:44:11,622 INFO [master/asf904:0:becomeActiveMaster] master.MasterFileSystem(393): BOOTSTRAP: creating hbase:meta region 2018-10-08 03:44:11,631 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(7043): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 Table name == hbase:meta 2018-10-08 03:44:11,677 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 32 2018-10-08 03:44:12,088 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(836): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:12,141 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/info 2018-10-08 03:44:12,163 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:12,180 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:12,199 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:12,204 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/rep_barrier 2018-10-08 03:44:12,206 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:12,214 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:12,216 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:12,221 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/table 2018-10-08 03:44:12,222 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:12,223 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:12,225 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:12,226 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(949): replaying wal for 1588230740 2018-10-08 03:44:12,240 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740 2018-10-08 03:44:12,242 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/meta/1588230740 2018-10-08 03:44:12,242 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(957): stopping wal replay for 1588230740 2018-10-08 03:44:12,243 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(969): Cleaning up temporary data for 1588230740 2018-10-08 03:44:12,253 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(980): Cleaning up detritus for 1588230740 2018-10-08 03:44:12,261 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-10-08 03:44:12,264 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1005): writing seq id for 1588230740 2018-10-08 03:44:12,272 DEBUG [master/asf904:0:becomeActiveMaster] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:12,273 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1009): Opened 1588230740; next sequenceid=2 2018-10-08 03:44:12,273 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1554): Closing 1588230740, disabling compactions & flushes 2018-10-08 03:44:12,274 DEBUG [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1594): Updates disabled for region hbase:meta,,1.1588230740 2018-10-08 03:44:12,276 INFO [master/asf904:0:becomeActiveMaster] regionserver.HRegion(1711): Closed hbase:meta,,1.1588230740 2018-10-08 03:44:12,319 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|FINALIZED]]} size 0 2018-10-08 03:44:12,326 DEBUG [master/asf904:0:becomeActiveMaster] util.FSTableDescriptors(683): Wrote into hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-10-08 03:44:12,374 INFO [master/asf904:0:becomeActiveMaster] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-10-08 03:44:12,391 INFO [master/asf904:0:becomeActiveMaster] coordination.ZKSplitLogManagerCoordination(494): Found 0 orphan tasks and 0 rescan nodes 2018-10-08 03:44:12,454 INFO [master/asf904:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x73f6806c to localhost:63142 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-08 03:44:12,500 DEBUG [master/asf904:0:becomeActiveMaster] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@99cbd5b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-08 03:44:12,545 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:12,555 DEBUG [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$DfsBuilderUtility(877): org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder not available, will not use builder API for file creation. 2018-10-08 03:44:12,560 WARN [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(994): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-10-08 03:44:12,560 DEBUG [master/asf904:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(1001): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:990) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:1028) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1072) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:418) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:658) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1498) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:938) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2294) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581) at java.lang.Thread.run(Thread.java:748) 2018-10-08 03:44:12,565 INFO [master/asf904:0:becomeActiveMaster] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=1 2018-10-08 03:44:12,568 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 20msec 2018-10-08 03:44:12,569 INFO [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 1msec 2018-10-08 03:44:12,569 INFO [master/asf904:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(97): Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-10-08 03:44:12,612 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(614): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-10-08 03:44:12,624 INFO [master/asf904:0:becomeActiveMaster] master.RegionServerTracker(123): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2018-10-08 03:44:12,674 INFO [master/asf904:0:becomeActiveMaster] balancer.BaseLoadBalancer(1039): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-10-08 03:44:12,684 INFO [master/asf904:0:becomeActiveMaster] balancer.StochasticLoadBalancer(216): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-10-08 03:44:12,713 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-10-08 03:44:12,714 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-10-08 03:44:12,719 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-10-08 03:44:12,721 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-10-08 03:44:12,757 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-10-08 03:44:12,758 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-10-08 03:44:12,758 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(796): Active/primary master=asf904.gq1.ygridcore.net,38583,1538970249417, sessionid=0x16651c665170000, setting cluster-up flag (Was=false) 2018-10-08 03:44:12,783 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-10-08 03:44:12,785 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:12,795 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-10-08 03:44:12,798 DEBUG [master/asf904:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:12,801 WARN [master/asf904:0:becomeActiveMaster] snapshot.SnapshotManager(283): Couldn't delete working snapshot directory: hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.hbase-snapshot/.tmp 2018-10-08 03:44:12,803 INFO [master/asf904:0:becomeActiveMaster] master.ServerManager(1112): No .lastflushedseqids found athdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.lastflushedseqids will record last flushed sequence id for regions by regionserver report all over again 2018-10-08 03:44:12,844 INFO [RS:0;asf904:47093] regionserver.HRegionServer(879): ClusterId : 312144ec-6fe1-49a3-857f-b64e3d5476c9 2018-10-08 03:44:12,851 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-10-08 03:44:12,861 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-10-08 03:44:12,861 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-10-08 03:44:12,865 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1011): hbase:meta {1588230740 state=OFFLINE, ts=1538970252615, server=null} 2018-10-08 03:44:12,867 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-10-08 03:44:12,868 INFO [RS:0;asf904:47093] zookeeper.ReadOnlyZKClient(139): Connect 0x0caab251 to localhost:63142 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-08 03:44:12,874 DEBUG [RS:0;asf904:47093] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26d5db4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-08 03:44:12,876 DEBUG [RS:0;asf904:47093] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@652d921, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf904.gq1.ygridcore.net/67.195.81.148:0 2018-10-08 03:44:12,880 DEBUG [RS:0;asf904:47093] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf904:47093 2018-10-08 03:44:12,885 INFO [RS:0;asf904:47093] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-10-08 03:44:12,885 INFO [RS:0;asf904:47093] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-10-08 03:44:12,888 INFO [RS:0;asf904:47093] regionserver.HRegionServer(2613): reportForDuty to master=asf904.gq1.ygridcore.net,38583,1538970249417 with port=47093, startcode=1538970250889 2018-10-08 03:44:12,937 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1124): Stored pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:12,959 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_OPEN_REGION-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-08 03:44:12,959 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_CLOSE_REGION-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-08 03:44:12,959 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_SERVER_OPERATIONS-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-08 03:44:12,960 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/asf904:0, corePoolSize=5, maxPoolSize=5 2018-10-08 03:44:12,960 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=M_LOG_REPLAY_OPS-master/asf904:0, corePoolSize=10, maxPoolSize=10 2018-10-08 03:44:12,960 DEBUG [master/asf904:0:becomeActiveMaster] executor.ExecutorService(98): Starting executor service name=MASTER_TABLE_OPERATIONS-master/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-08 03:44:12,962 INFO [master/asf904:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970282961 2018-10-08 03:44:12,966 INFO [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore$DirScanPool(90): Cleaner pool size is 4 2018-10-08 03:44:12,968 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-10-08 03:44:12,969 INFO [master/asf904:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:63142 2018-10-08 03:44:12,971 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-10-08 03:44:12,974 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-10-08 03:44:12,974 INFO [master/asf904:0:becomeActiveMaster] cleaner.LogCleaner(155): Creating OldWALs cleaners with size=2 2018-10-08 03:44:12,980 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner0x0, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-10-08 03:44:12,980 DEBUG [master/asf904:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(543): replicationLogCleaner-0x16651c665170004 connected 2018-10-08 03:44:12,985 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-10-08 03:44:12,989 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-10-08 03:44:12,990 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-10-08 03:44:12,993 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.HFileCleaner(225): Starting for large file=Thread[master/asf904:0:becomeActiveMaster-HFileCleaner.large.0-1538970252993,5,FailOnTimeoutGroup] 2018-10-08 03:44:12,994 DEBUG [master/asf904:0:becomeActiveMaster] cleaner.HFileCleaner(240): Starting for small files=Thread[master/asf904:0:becomeActiveMaster-HFileCleaner.small.0-1538970252994,5,FailOnTimeoutGroup] 2018-10-08 03:44:13,007 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2018-10-08 03:44:13,029 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN checking lock on 1588230740 2018-10-08 03:44:13,031 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:50873, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2018-10-08 03:44:13,033 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-10-08 03:44:13,053 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.ServerManager(439): Registering regionserver=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,078 DEBUG [RS:0;asf904:47093] regionserver.HRegionServer(1506): Config from master: hbase.rootdir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:13,079 DEBUG [RS:0;asf904:47093] regionserver.HRegionServer(1506): Config from master: fs.defaultFS=hdfs://localhost:54583 2018-10-08 03:44:13,079 DEBUG [RS:0;asf904:47093] regionserver.HRegionServer(1506): Config from master: hbase.master.info.port=-1 2018-10-08 03:44:13,087 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-08 03:44:13,088 DEBUG [RS:0;asf904:47093] zookeeper.ZKUtil(355): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,088 WARN [RS:0;asf904:47093] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-10-08 03:44:13,091 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node created, adding [asf904.gq1.ygridcore.net,47093,1538970250889] 2018-10-08 03:44:13,113 DEBUG [RS:0;asf904:47093] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-10-08 03:44:13,156 DEBUG [RS:0;asf904:47093] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:128) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:199) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1814) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1528) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:157) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:958) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) 2018-10-08 03:44:13,158 DEBUG [RS:0;asf904:47093] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-10-08 03:44:13,163 DEBUG [RS:0;asf904:47093] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(196): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:194) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:299) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:129) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:199) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1814) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1528) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:157) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:958) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) 2018-10-08 03:44:13,164 INFO [RS:0;asf904:47093] wal.WALFactory(157): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-10-08 03:44:13,172 DEBUG [RS:0;asf904:47093] regionserver.HRegionServer(1821): logDir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,187 DEBUG [RS:0;asf904:47093] zookeeper.ZKUtil(355): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,187 DEBUG [master/asf904:0] assignment.AssignmentManager(1739): Processing assignQueue; systemServersCount=1, allServersCount=1 2018-10-08 03:44:13,189 INFO [PEWorker-1] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,197 DEBUG [RS:0;asf904:47093] regionserver.Replication(144): Replication stats-in-log period=300 seconds 2018-10-08 03:44:13,200 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-10-08 03:44:13,204 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:13,210 INFO [RS:0;asf904:47093] regionserver.MetricsRegionServerWrapperImpl(147): Computing regionserver metrics every 5000 milliseconds 2018-10-08 03:44:13,237 INFO [RS:0;asf904:47093] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-10-08 03:44:13,244 INFO [RS:0;asf904:47093] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-10-08 03:44:13,247 INFO [RS:0;asf904:47093] regionserver.HRegionServer$CompactionChecker(1707): CompactionChecker runs every PT1S 2018-10-08 03:44:13,262 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_OPEN_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-08 03:44:13,263 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_OPEN_META-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-08 03:44:13,263 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-08 03:44:13,263 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_CLOSE_REGION-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-08 03:44:13,263 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_CLOSE_META-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-08 03:44:13,264 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf904:0, corePoolSize=2, maxPoolSize=2 2018-10-08 03:44:13,264 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0, corePoolSize=10, maxPoolSize=10 2018-10-08 03:44:13,264 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf904:0, corePoolSize=3, maxPoolSize=3 2018-10-08 03:44:13,264 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_REFRESH_PEER-regionserver/asf904:0, corePoolSize=2, maxPoolSize=2 2018-10-08 03:44:13,264 DEBUG [RS:0;asf904:47093] executor.ExecutorService(98): Starting executor service name=RS_REPLAY_SYNC_REPLICATION_WAL-regionserver/asf904:0, corePoolSize=1, maxPoolSize=1 2018-10-08 03:44:13,291 INFO [SplitLogWorker-asf904:47093] regionserver.SplitLogWorker(211): SplitLogWorker asf904.gq1.ygridcore.net,47093,1538970250889 starting 2018-10-08 03:44:13,295 INFO [RS:0;asf904:47093] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-10-08 03:44:13,332 INFO [RS:0;asf904:47093] regionserver.HRegionServer(1547): Serving as asf904.gq1.ygridcore.net,47093,1538970250889, RpcServer on asf904.gq1.ygridcore.net/67.195.81.148:47093, sessionid=0x16651c665170001 2018-10-08 03:44:13,333 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-10-08 03:44:13,333 DEBUG [RS:0;asf904:47093] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,333 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf904.gq1.ygridcore.net,47093,1538970250889' 2018-10-08 03:44:13,333 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-10-08 03:44:13,334 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-10-08 03:44:13,335 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-10-08 03:44:13,335 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-10-08 03:44:13,335 DEBUG [RS:0;asf904:47093] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,335 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf904.gq1.ygridcore.net,47093,1538970250889' 2018-10-08 03:44:13,335 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-10-08 03:44:13,336 DEBUG [RS:0;asf904:47093] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-10-08 03:44:13,337 DEBUG [RS:0;asf904:47093] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-10-08 03:44:13,337 INFO [RS:0;asf904:47093] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-10-08 03:44:13,337 INFO [RS:0;asf904:47093] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-10-08 03:44:13,420 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(746): New admin connection to asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,433 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:55064, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2018-10-08 03:44:13,441 INFO [RS_CLOSE_META-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open hbase:meta,,1.1588230740 2018-10-08 03:44:13,442 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.WALFactory(157): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-10-08 03:44:13,467 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(419): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf904.gq1.ygridcore.net%2C47093%2C1538970250889.meta, suffix=.meta, logDir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889, archiveDir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/oldWALs 2018-10-08 03:44:13,508 DEBUG [RS-EventLoopGroup-3-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41173,DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3,DISK] 2018-10-08 03:44:13,541 INFO [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(684): New WAL /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889/asf904.gq1.ygridcore.net%2C47093%2C1538970250889.meta.1538970253488.meta 2018-10-08 03:44:13,542 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] wal.AbstractFSWAL(773): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41173,DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3,DISK]] 2018-10-08 03:44:13,543 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-10-08 03:44:13,578 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-10-08 03:44:13,606 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(8196): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-10-08 03:44:13,616 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-10-08 03:44:13,623 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-10-08 03:44:13,623 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:13,624 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 1588230740 2018-10-08 03:44:13,625 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 1588230740 2018-10-08 03:44:13,636 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/info 2018-10-08 03:44:13,636 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/info 2018-10-08 03:44:13,638 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:13,640 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:13,642 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:13,648 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/rep_barrier 2018-10-08 03:44:13,648 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/rep_barrier 2018-10-08 03:44:13,650 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:13,650 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:13,652 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:13,655 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/table 2018-10-08 03:44:13,656 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/table 2018-10-08 03:44:13,656 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:13,657 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:13,663 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:13,664 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 1588230740 2018-10-08 03:44:13,667 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740 2018-10-08 03:44:13,672 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/meta/1588230740 2018-10-08 03:44:13,672 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 1588230740 2018-10-08 03:44:13,672 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 1588230740 2018-10-08 03:44:13,674 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 1588230740 2018-10-08 03:44:13,677 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-10-08 03:44:13,678 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 1588230740 2018-10-08 03:44:13,680 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 1588230740; next sequenceid=2 2018-10-08 03:44:13,680 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 1588230740 2018-10-08 03:44:13,730 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionServer(2198): Post open deploy tasks for hbase:meta,,1.1588230740 2018-10-08 03:44:13,754 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=38583] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:13,757 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-10-08 03:44:13,761 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegionServer(2222): Finished post open deploy task for hbase:meta,,1.1588230740 2018-10-08 03:44:13,763 INFO [RS_CLOSE_META-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened hbase:meta,,1.1588230740 2018-10-08 03:44:13,772 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; resume parent processing. 2018-10-08 03:44:13,772 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=3, ppid=2, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 559msec 2018-10-08 03:44:13,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=1, state=RUNNABLE, hasLock=false; InitMetaProcedure table=hbase:meta; resume parent processing. 2018-10-08 03:44:13,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=2, ppid=1, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 767msec 2018-10-08 03:44:13,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta in 912msec 2018-10-08 03:44:13,791 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1047): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1538970250980, completionTime=-1 2018-10-08 03:44:13,791 INFO [master/asf904:0:becomeActiveMaster] master.ServerManager(854): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2018-10-08 03:44:13,791 DEBUG [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:13,885 INFO [RS-EventLoopGroup-3-8] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:55070, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-08 03:44:13,928 INFO [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:13,929 INFO [master/asf904:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970313929 2018-10-08 03:44:13,930 INFO [master/asf904:0:becomeActiveMaster] assignment.AssignmentManager(1234): Joined the cluster in 138msec 2018-10-08 03:44:14,012 INFO [master/asf904:0:becomeActiveMaster] master.TableNamespaceManager(96): Namespace table not found. Creating... 2018-10-08 03:44:14,018 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(2040): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-10-08 03:44:14,034 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1124): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:14,054 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. from FS 2018-10-08 03:44:14,057 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:14,060 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3 empty. 2018-10-08 03:44:14,062 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:14,062 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'hbase:namespace' archived! 2018-10-08 03:44:14,095 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741831_1007{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 476 2018-10-08 03:44:14,357 INFO [RS:0;asf904:47093] wal.AbstractFSWAL(419): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf904.gq1.ygridcore.net%2C47093%2C1538970250889, suffix=, logDir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889, archiveDir=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/oldWALs 2018-10-08 03:44:14,371 DEBUG [RS-EventLoopGroup-3-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41173,DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a,DISK] 2018-10-08 03:44:14,379 INFO [RS:0;asf904:47093] wal.AbstractFSWAL(684): New WAL /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889/asf904.gq1.ygridcore.net%2C47093%2C1538970250889.1538970254360 2018-10-08 03:44:14,379 DEBUG [RS:0;asf904:47093] wal.AbstractFSWAL(773): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41173,DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a,DISK]] 2018-10-08 03:44:14,501 DEBUG [PEWorker-1] util.FSTableDescriptors(683): Wrote into hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-10-08 03:44:14,505 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(7043): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == hbase:namespace 2018-10-08 03:44:14,531 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741833_1009{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 42 2018-10-08 03:44:14,934 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(836): Instantiated hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:14,935 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1554): Closing e298524ff1440c61e0c0af4e0880daf3, disabling compactions & flushes 2018-10-08 03:44:14,935 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1594): Updates disabled for region hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:14,935 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1711): Closed hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:15,006 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":1538970254948},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970254948}]},"ts":1538970254948} 2018-10-08 03:44:15,056 INFO [PEWorker-1] hbase.MetaTableAccessor(1555): Added 1 regions to meta. 2018-10-08 03:44:15,066 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970255060}]},"ts":1538970255060} 2018-10-08 03:44:15,073 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2018-10-08 03:44:15,081 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=e298524ff1440c61e0c0af4e0880daf3, ASSIGN}] 2018-10-08 03:44:15,088 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=e298524ff1440c61e0c0af4e0880daf3, ASSIGN checking lock on e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,089 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=e298524ff1440c61e0c0af4e0880daf3, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:15,241 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=5 updating hbase:meta row=e298524ff1440c61e0c0af4e0880daf3, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:15,250 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:15,424 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:15,429 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => e298524ff1440c61e0c0af4e0880daf3, NAME => 'hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3.', STARTKEY => '', ENDKEY => ''} 2018-10-08 03:44:15,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:15,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,442 DEBUG [StoreOpener-e298524ff1440c61e0c0af4e0880daf3-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/info 2018-10-08 03:44:15,443 DEBUG [StoreOpener-e298524ff1440c61e0c0af4e0880daf3-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/info 2018-10-08 03:44:15,446 INFO [StoreOpener-e298524ff1440c61e0c0af4e0880daf3-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:15,447 INFO [StoreOpener-e298524ff1440c61e0c0af4e0880daf3-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:15,449 INFO [StoreOpener-e298524ff1440c61e0c0af4e0880daf3-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:15,450 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,454 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,458 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,462 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,472 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:15,472 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened e298524ff1440c61e0c0af4e0880daf3; next sequenceid=2 2018-10-08 03:44:15,474 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for e298524ff1440c61e0c0af4e0880daf3 2018-10-08 03:44:15,479 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2198): Post open deploy tasks for hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:15,492 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=38583] assignment.RegionStateStore(200): pid=5 updating hbase:meta row=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:15,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2222): Finished post open deploy task for hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:15,508 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:44:15,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=e298524ff1440c61e0c0af4e0880daf3, ASSIGN; resume parent processing. 2018-10-08 03:44:15,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=6, ppid=5, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 259msec 2018-10-08 03:44:15,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, hasLock=false; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-10-08 03:44:15,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=5, ppid=4, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=e298524ff1440c61e0c0af4e0880daf3, ASSIGN in 433msec 2018-10-08 03:44:15,523 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970255523}]},"ts":1538970255523} 2018-10-08 03:44:15,533 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2018-10-08 03:44:15,539 DEBUG [master/asf904:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-10-08 03:44:15,540 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-10-08 03:44:15,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace in 1.5160sec 2018-10-08 03:44:15,599 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1124): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:15,635 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-10-08 03:44:15,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default in 53msec 2018-10-08 03:44:15,649 DEBUG [master/asf904:0:becomeActiveMaster] procedure2.ProcedureExecutor(1124): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:15,671 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-10-08 03:44:15,677 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase in 27msec 2018-10-08 03:44:15,689 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-10-08 03:44:15,691 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-10-08 03:44:15,691 INFO [master/asf904:0:becomeActiveMaster] master.HMaster(1110): Master has completed initialization 4.696sec 2018-10-08 03:44:15,695 INFO [master/asf904:0:becomeActiveMaster] quotas.MasterQuotaManager(90): Quota support disabled 2018-10-08 03:44:15,695 INFO [master/asf904:0:becomeActiveMaster] zookeeper.ZKWatcher(205): not a secure deployment, proceeding 2018-10-08 03:44:15,710 DEBUG [master/asf904:0:becomeActiveMaster] master.HMaster(1168): Balancer post startup initialization complete, took 0 seconds 2018-10-08 03:44:15,752 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x4431d292 to localhost:63142 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-10-08 03:44:15,760 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f6262f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-10-08 03:44:15,793 INFO [RS-EventLoopGroup-3-11] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:55139, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-08 03:44:15,813 INFO [Time-limited test] hbase.HBaseTestingUtility(1103): Minicluster is up; activeMaster=asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:44:15,868 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testMergeWithoutPONR Thread=197, OpenFileDescriptor=848, MaxFileDescriptor=60000, SystemLoadAverage=602, ProcessCount=289, AvailableMemoryMB=10675 2018-10-08 03:44:15,873 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:15,896 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:32894, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2018-10-08 03:44:15,916 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2018-10-08 03:44:15,918 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(422): Client=jenkins//67.195.81.148 set balanceSwitch=false 2018-10-08 03:44:15,922 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:15,940 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.HMaster$3(2004): Client=jenkins//67.195.81.148 create 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-10-08 03:44:15,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] procedure2.ProcedureExecutor(1124): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:15,956 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(630): Client=jenkins//67.195.81.148 procedure request for creating table: namespace: "default" qualifier: "testMergeWithoutPONR" procId is: 9 2018-10-08 03:44:15,956 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. from FS 2018-10-08 03:44:15,957 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:15,959 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6 empty. 2018-10-08 03:44:15,962 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:15,962 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. from FS 2018-10-08 03:44:15,962 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:15,964 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec empty. 2018-10-08 03:44:15,965 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:15,965 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. from FS 2018-10-08 03:44:15,965 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:15,967 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7 empty. 2018-10-08 03:44:15,968 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:15,968 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. from FS 2018-10-08 03:44:15,968 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:15,970 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801 empty. 2018-10-08 03:44:15,973 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:15,974 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'testMergeWithoutPONR' archived! 2018-10-08 03:44:15,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:16,000 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 492 2018-10-08 03:44:16,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:16,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:16,405 DEBUG [PEWorker-1] util.FSTableDescriptors(683): Wrote into hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/.tabledesc/.tableinfo.0000000001 2018-10-08 03:44:16,409 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testMergeWithoutPONR 2018-10-08 03:44:16,410 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testMergeWithoutPONR 2018-10-08 03:44:16,411 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testMergeWithoutPONR 2018-10-08 03:44:16,412 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(7043): creating HRegion testMergeWithoutPONR HTD == 'testMergeWithoutPONR', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testMergeWithoutPONR 2018-10-08 03:44:16,442 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741835_1011{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 56 2018-10-08 03:44:16,446 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741836_1012{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 56 2018-10-08 03:44:16,447 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741837_1013{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 57 2018-10-08 03:44:16,454 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741838_1014{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 57 2018-10-08 03:44:16,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:16,846 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:16,847 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1554): Closing 7beb91c361da5530dbf69d1615f54801, disabling compactions & flushes 2018-10-08 03:44:16,847 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:16,847 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:16,847 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1554): Closing 7b2f6b48e0a7b7e4802db18892a95ae7, disabling compactions & flushes 2018-10-08 03:44:16,847 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-4] regionserver.HRegion(1711): Closed testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:16,848 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:16,848 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:16,848 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-3] regionserver.HRegion(1711): Closed testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:16,848 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1554): Closing 0a1f839303f18eb89f2dd2858e8f53e6, disabling compactions & flushes 2018-10-08 03:44:16,848 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:16,848 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:16,856 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:16,856 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1554): Closing 64b6252f0678baac0ab16d1191421dec, disabling compactions & flushes 2018-10-08 03:44:16,856 DEBUG [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:16,856 INFO [RegionOpenAndInitThread-testMergeWithoutPONR-2] regionserver.HRegion(1711): Closed testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:16,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":1538970256868},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970256868}]},"ts":1538970256868} 2018-10-08 03:44:16,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1538970256868},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970256868}]},"ts":1538970256868} 2018-10-08 03:44:16,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":1538970256868},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970256868}]},"ts":1538970256868} 2018-10-08 03:44:16,869 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1538970256868},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970256868}]},"ts":1538970256868} 2018-10-08 03:44:16,946 INFO [PEWorker-1] hbase.MetaTableAccessor(1555): Added 4 regions to meta. 2018-10-08 03:44:16,950 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970256949}]},"ts":1538970256949} 2018-10-08 03:44:16,955 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=ENABLING in hbase:meta 2018-10-08 03:44:16,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, ASSIGN}] 2018-10-08 03:44:16,965 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, ASSIGN checking lock on 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:16,967 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:16,969 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, ASSIGN checking lock on 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:16,970 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:16,972 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, ASSIGN checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:16,974 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:16,979 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, ASSIGN checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:16,981 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:17,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:17,121 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=11 updating hbase:meta row=64b6252f0678baac0ab16d1191421dec, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,128 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=14, ppid=11, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:17,142 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=10 updating hbase:meta row=0a1f839303f18eb89f2dd2858e8f53e6, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=15, ppid=10, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:17,155 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=13 updating hbase:meta row=7beb91c361da5530dbf69d1615f54801, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,162 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=16, ppid=13, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:17,169 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=12 updating hbase:meta row=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:17,300 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:17,301 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 0a1f839303f18eb89f2dd2858e8f53e6, NAME => 'testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6.', STARTKEY => '', ENDKEY => '0'} 2018-10-08 03:44:17,301 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,302 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:17,302 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,302 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,306 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:17,307 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7217): Opening region: {ENCODED => 64b6252f0678baac0ab16d1191421dec, NAME => 'testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec.', STARTKEY => '0', ENDKEY => '1'} 2018-10-08 03:44:17,307 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,307 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:17,307 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7256): checking encryption for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,307 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7261): checking classloading for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,310 DEBUG [StoreOpener-0a1f839303f18eb89f2dd2858e8f53e6-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6/FAMILY 2018-10-08 03:44:17,310 DEBUG [StoreOpener-0a1f839303f18eb89f2dd2858e8f53e6-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6/FAMILY 2018-10-08 03:44:17,313 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:17,314 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7217): Opening region: {ENCODED => 7b2f6b48e0a7b7e4802db18892a95ae7, NAME => 'testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.', STARTKEY => '1', ENDKEY => '2'} 2018-10-08 03:44:17,315 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,317 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:17,318 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7256): checking encryption for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,318 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7261): checking classloading for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,318 DEBUG [StoreOpener-64b6252f0678baac0ab16d1191421dec-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec/FAMILY 2018-10-08 03:44:17,318 DEBUG [StoreOpener-64b6252f0678baac0ab16d1191421dec-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec/FAMILY 2018-10-08 03:44:17,323 INFO [StoreOpener-64b6252f0678baac0ab16d1191421dec-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:17,323 INFO [StoreOpener-64b6252f0678baac0ab16d1191421dec-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:17,323 INFO [StoreOpener-0a1f839303f18eb89f2dd2858e8f53e6-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:17,327 INFO [StoreOpener-0a1f839303f18eb89f2dd2858e8f53e6-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:17,328 INFO [StoreOpener-64b6252f0678baac0ab16d1191421dec-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:17,329 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(949): replaying wal for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,332 INFO [StoreOpener-0a1f839303f18eb89f2dd2858e8f53e6-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:17,332 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,335 DEBUG [StoreOpener-7b2f6b48e0a7b7e4802db18892a95ae7-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7/FAMILY 2018-10-08 03:44:17,335 DEBUG [StoreOpener-7b2f6b48e0a7b7e4802db18892a95ae7-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7/FAMILY 2018-10-08 03:44:17,336 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,336 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,337 INFO [StoreOpener-7b2f6b48e0a7b7e4802db18892a95ae7-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:17,337 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,337 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,338 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(957): stopping wal replay for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,339 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(969): Cleaning up temporary data for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,337 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,338 INFO [StoreOpener-7b2f6b48e0a7b7e4802db18892a95ae7-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:17,339 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,344 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,344 INFO [StoreOpener-7b2f6b48e0a7b7e4802db18892a95ae7-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:17,345 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(949): replaying wal for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,346 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(980): Cleaning up detritus for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,348 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,349 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,350 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,350 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(957): stopping wal replay for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,350 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1005): writing seq id for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,351 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(969): Cleaning up temporary data for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,352 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(980): Cleaning up detritus for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,357 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:17,358 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 0a1f839303f18eb89f2dd2858e8f53e6; next sequenceid=2 2018-10-08 03:44:17,358 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:17,358 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1005): writing seq id for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,359 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:17,359 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1009): Opened 64b6252f0678baac0ab16d1191421dec; next sequenceid=2 2018-10-08 03:44:17,359 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1016): Running coprocessor post-open hooks for 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:17,362 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2198): Post open deploy tasks for testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:17,362 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2198): Post open deploy tasks for testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:17,368 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:17,368 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1009): Opened 7b2f6b48e0a7b7e4802db18892a95ae7; next sequenceid=2 2018-10-08 03:44:17,368 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1016): Running coprocessor post-open hooks for 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:17,371 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2198): Post open deploy tasks for testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:17,378 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=11 updating hbase:meta row=64b6252f0678baac0ab16d1191421dec, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,378 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=38583] assignment.RegionStateStore(200): pid=10 updating hbase:meta row=0a1f839303f18eb89f2dd2858e8f53e6, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] assignment.RegionStateStore(200): pid=12 updating hbase:meta row=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,389 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2222): Finished post open deploy task for testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:17,389 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:17,389 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2222): Finished post open deploy task for testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:17,389 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:17,389 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2222): Finished post open deploy task for testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:17,389 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:17,389 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:17,390 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7217): Opening region: {ENCODED => 7beb91c361da5530dbf69d1615f54801, NAME => 'testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.', STARTKEY => '2', ENDKEY => ''} 2018-10-08 03:44:17,390 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,391 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:17,391 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7256): checking encryption for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,391 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7261): checking classloading for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, ASSIGN; resume parent processing. 2018-10-08 03:44:17,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=14, ppid=11, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 264msec 2018-10-08 03:44:17,399 DEBUG [StoreOpener-7beb91c361da5530dbf69d1615f54801-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801/FAMILY 2018-10-08 03:44:17,399 DEBUG [StoreOpener-7beb91c361da5530dbf69d1615f54801-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801/FAMILY 2018-10-08 03:44:17,400 INFO [StoreOpener-7beb91c361da5530dbf69d1615f54801-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:17,400 INFO [StoreOpener-7beb91c361da5530dbf69d1615f54801-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:17,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=11, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, ASSIGN in 439msec 2018-10-08 03:44:17,404 INFO [StoreOpener-7beb91c361da5530dbf69d1615f54801-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:17,405 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(949): replaying wal for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,408 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,409 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,410 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(957): stopping wal replay for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,410 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(969): Cleaning up temporary data for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,411 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(980): Cleaning up detritus for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, ASSIGN; resume parent processing. 2018-10-08 03:44:17,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=17, ppid=12, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 232msec 2018-10-08 03:44:17,413 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1005): writing seq id for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=12, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, ASSIGN in 454msec 2018-10-08 03:44:17,419 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:17,419 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1009): Opened 7beb91c361da5530dbf69d1615f54801; next sequenceid=2 2018-10-08 03:44:17,419 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1016): Running coprocessor post-open hooks for 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:17,421 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2198): Post open deploy tasks for testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:17,422 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, ASSIGN; resume parent processing. 2018-10-08 03:44:17,422 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=15, ppid=10, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 270msec 2018-10-08 03:44:17,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] assignment.RegionStateStore(200): pid=13 updating hbase:meta row=7beb91c361da5530dbf69d1615f54801, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:17,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=10, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, ASSIGN in 463msec 2018-10-08 03:44:17,429 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2222): Finished post open deploy task for testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:17,429 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:17,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, ASSIGN; resume parent processing. 2018-10-08 03:44:17,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=16, ppid=13, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 269msec 2018-10-08 03:44:17,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR; resume parent processing. 2018-10-08 03:44:17,439 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=13, ppid=9, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, ASSIGN in 476msec 2018-10-08 03:44:17,441 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970257441}]},"ts":1538970257441} 2018-10-08 03:44:17,445 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=ENABLED in hbase:meta 2018-10-08 03:44:17,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR in 1.5090sec 2018-10-08 03:44:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=9 2018-10-08 03:44:18,129 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3721): Operation: CREATE, Table Name: default:testMergeWithoutPONR, procId: 9 completed 2018-10-08 03:44:18,134 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [900,000] milli-secs(wait.for.ratio=[1]) 2018-10-08 03:44:18,404 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: true 2018-10-08 03:44:18,424 DEBUG [Time-limited test] procedure2.ProcedureExecutor(1124): Stored pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true 2018-10-08 03:44:18,425 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:18,425 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:18,426 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:18,459 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:18,459 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true 2018-10-08 03:44:18,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:18,460 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:18,675 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(404): Restart 3 exec state=pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=true 2018-10-08 03:44:18,677 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:18,677 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:18,695 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741829_1005{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 56917 2018-10-08 03:44:19,098 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:19,100 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:19,101 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:19,104 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:19,115 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 11ms 2018-10-08 03:44:19,116 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:19,134 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=2 2018-10-08 03:44:19,136 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 34msec 2018-10-08 03:44:19,142 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-08 03:44:19,169 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:19,173 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:19,173 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:19,174 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:19,174 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:19,175 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:19,175 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,176 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:19,176 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:19,178 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:19,178 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:19,180 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 43msec 2018-10-08 03:44:19,183 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970289183 2018-10-08 03:44:19,185 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,185 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:19,209 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,210 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=0a1f839303f18eb89f2dd2858e8f53e6, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,211 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=64b6252f0678baac0ab16d1191421dec, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,211 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,212 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7beb91c361da5530dbf69d1615f54801, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,213 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:19,213 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970319213 2018-10-08 03:44:19,213 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 28msec 2018-10-08 03:44:19,231 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:19,266 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:19,266 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,266 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:19,267 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:19,464 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(404): Restart 4 exec state=pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,464 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:19,464 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:19,473 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741839_1015{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 470 2018-10-08 03:44:19,587 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-10-08 03:44:19,587 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-10-08 03:44:19,880 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:19,881 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:19,881 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:19,884 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:19,884 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 0ms 2018-10-08 03:44:19,885 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:19,892 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:19,893 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:19,893 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:19,904 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=3 2018-10-08 03:44:19,906 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 24msec 2018-10-08 03:44:19,911 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:19,924 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:19,927 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:19,928 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:19,928 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:19,929 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:19,929 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:19,929 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,929 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:19,930 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:19,930 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:19,930 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:19,931 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 24msec 2018-10-08 03:44:19,932 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970289932 2018-10-08 03:44:19,933 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:19,936 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,950 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,951 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=0a1f839303f18eb89f2dd2858e8f53e6, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,951 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=64b6252f0678baac0ab16d1191421dec, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,952 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,952 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7beb91c361da5530dbf69d1615f54801, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:19,952 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:19,953 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970319953 2018-10-08 03:44:19,953 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 19msec 2018-10-08 03:44:19,953 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:19,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, UNASSIGN}] 2018-10-08 03:44:19,973 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:19,973 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:19,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:19,973 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:20,203 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:20,203 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:20,204 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:20,210 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 474 2018-10-08 03:44:20,613 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:20,614 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:20,615 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:20,617 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:20,619 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-10-08 03:44:20,619 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:20,627 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:20,628 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:20,629 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:20,637 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:20,640 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 2ms 2018-10-08 03:44:20,640 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:20,650 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=4 2018-10-08 03:44:20,652 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 36msec 2018-10-08 03:44:20,656 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:20,661 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:20,671 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:20,673 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:20,673 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:20,673 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:20,674 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:20,674 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:20,674 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:20,675 DEBUG [Time-limited test] procedure2.Procedure(979): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:20,675 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:20,675 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:20,675 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:20,676 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 23msec 2018-10-08 03:44:20,677 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970290677 2018-10-08 03:44:20,678 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:20,679 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:20,686 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:20,687 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=0a1f839303f18eb89f2dd2858e8f53e6, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:20,688 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=64b6252f0678baac0ab16d1191421dec, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:20,689 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:20,689 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=7beb91c361da5530dbf69d1615f54801, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:20,690 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:20,690 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970320690 2018-10-08 03:44:20,690 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 12msec 2018-10-08 03:44:20,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, UNASSIGN}] 2018-10-08 03:44:20,705 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, UNASSIGN checking lock on 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:20,707 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=20 updating hbase:meta row=64b6252f0678baac0ab16d1191421dec, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:20,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:20,723 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, UNASSIGN checking lock on 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:20,724 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=0a1f839303f18eb89f2dd2858e8f53e6, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:20,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:20,879 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(102): Close 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:20,879 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(102): Close 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:20,881 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1554): Closing 64b6252f0678baac0ab16d1191421dec, disabling compactions & flushes 2018-10-08 03:44:20,881 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:20,882 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 0a1f839303f18eb89f2dd2858e8f53e6, disabling compactions & flushes 2018-10-08 03:44:20,882 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:20,898 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/0a1f839303f18eb89f2dd2858e8f53e6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:20,898 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/64b6252f0678baac0ab16d1191421dec/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:20,905 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec. 2018-10-08 03:44:20,905 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6. 2018-10-08 03:44:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] assignment.RegionStateStore(200): pid=19 updating hbase:meta row=0a1f839303f18eb89f2dd2858e8f53e6, regionState=CLOSED 2018-10-08 03:44:20,907 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=20 updating hbase:meta row=64b6252f0678baac0ab16d1191421dec, regionState=CLOSED 2018-10-08 03:44:20,921 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(124): Closed 64b6252f0678baac0ab16d1191421dec 2018-10-08 03:44:20,921 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(124): Closed 0a1f839303f18eb89f2dd2858e8f53e6 2018-10-08 03:44:20,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, UNASSIGN; resume parent processing. 2018-10-08 03:44:20,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=21, ppid=20, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 210msec 2018-10-08 03:44:20,934 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=20, ppid=18, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=64b6252f0678baac0ab16d1191421dec, UNASSIGN in 236msec 2018-10-08 03:44:20,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, UNASSIGN; resume parent processing. 2018-10-08 03:44:20,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=22, ppid=19, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 207msec 2018-10-08 03:44:20,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false; resume parent processing. 2018-10-08 03:44:20,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=19, ppid=18, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=0a1f839303f18eb89f2dd2858e8f53e6, UNASSIGN in 250msec 2018-10-08 03:44:20,996 DEBUG [PEWorker-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2018-10-08 03:44:21,002 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,,1538970255933.0a1f839303f18eb89f2dd2858e8f53e6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970261001}]},"ts":9223372036854775807} 2018-10-08 03:44:21,002 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,0,1538970255933.64b6252f0678baac0ab16d1191421dec.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970261001}]},"ts":9223372036854775807} 2018-10-08 03:44:21,003 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":7,"row":"testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.","families":{"info":[{"qualifier":"regioninfo","vlen":55,"tag":[],"timestamp":1538970261001},{"qualifier":"mergeA","vlen":55,"tag":[],"timestamp":1538970261001},{"qualifier":"mergeB","vlen":56,"tag":[],"timestamp":1538970261001},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970261001}]},"ts":1538970261001} 2018-10-08 03:44:21,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, ASSIGN}] 2018-10-08 03:44:21,070 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, ASSIGN checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,072 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, ASSIGN; rit=MERGED, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:21,224 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=23 updating hbase:meta row=13dce85b09b0d1a075f661c037edf834, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:21,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:21,391 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:21,392 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 13dce85b09b0d1a075f661c037edf834, NAME => 'testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.', STARTKEY => '', ENDKEY => '1'} 2018-10-08 03:44:21,392 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMergeWithoutPONR 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,392 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:21,393 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,393 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,394 WARN [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionFileSystem(889): hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834 doesn't exist for region: 13dce85b09b0d1a075f661c037edf834 on table testMergeWithoutPONR 2018-10-08 03:44:21,395 WARN [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionFileSystem(907): .regioninfo file not found for region: 13dce85b09b0d1a075f661c037edf834 on table testMergeWithoutPONR 2018-10-08 03:44:21,410 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741842_1018{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 56 2018-10-08 03:44:21,819 DEBUG [StoreOpener-13dce85b09b0d1a075f661c037edf834-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834/FAMILY 2018-10-08 03:44:21,819 DEBUG [StoreOpener-13dce85b09b0d1a075f661c037edf834-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834/FAMILY 2018-10-08 03:44:21,821 INFO [StoreOpener-13dce85b09b0d1a075f661c037edf834-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:21,821 INFO [StoreOpener-13dce85b09b0d1a075f661c037edf834-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:21,823 INFO [StoreOpener-13dce85b09b0d1a075f661c037edf834-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:21,823 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,826 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,829 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,830 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,830 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,831 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,834 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,835 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 13dce85b09b0d1a075f661c037edf834; next sequenceid=5 2018-10-08 03:44:21,835 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:21,837 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2198): Post open deploy tasks for testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:21,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] assignment.RegionStateStore(200): pid=23 updating hbase:meta row=13dce85b09b0d1a075f661c037edf834, regionState=OPEN, openSeqNum=5, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:21,846 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2222): Finished post open deploy task for testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:21,846 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:21,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=23, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, ASSIGN; resume parent processing. 2018-10-08 03:44:21,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=24, ppid=23, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 618msec 2018-10-08 03:44:21,854 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=18, state=RUNNABLE:MERGE_TABLE_REGIONS_POST_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false; resume parent processing. 2018-10-08 03:44:21,854 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=23, ppid=18, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, ASSIGN in 783msec 2018-10-08 03:44:21,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false in 3.4430sec 2018-10-08 03:44:21,941 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [900,000] milli-secs(wait.for.ratio=[1]) 2018-10-08 03:44:21,952 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:21,965 INFO [Time-limited test] assignment.TestMergeTableRegionsProcedure(129): Tear down, remove table=testMergeWithoutPONR 2018-10-08 03:44:21,968 INFO [Time-limited test] client.HBaseAdmin$15(922): Started disable of testMergeWithoutPONR 2018-10-08 03:44:21,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] master.HMaster$10(2554): Client=jenkins//67.195.81.148 disable testMergeWithoutPONR 2018-10-08 03:44:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] procedure2.ProcedureExecutor(1124): Stored pid=25, state=RUNNABLE:DISABLE_TABLE_PREPARE, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:21,999 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970261999}]},"ts":1538970261999} 2018-10-08 03:44:22,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=25 2018-10-08 03:44:22,003 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=DISABLING in hbase:meta 2018-10-08 03:44:22,006 INFO [PEWorker-1] procedure.DisableTableProcedure(295): Set testMergeWithoutPONR to state=DISABLING 2018-10-08 03:44:22,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, UNASSIGN}, {pid=27, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, UNASSIGN}, {pid=28, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, UNASSIGN}] 2018-10-08 03:44:22,014 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=28, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, UNASSIGN checking lock on 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:22,015 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=28 updating hbase:meta row=7beb91c361da5530dbf69d1615f54801, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:22,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:22,024 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=27, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, UNASSIGN checking lock on 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:22,025 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=27 updating hbase:meta row=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:22,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=30, ppid=27, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:22,034 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, UNASSIGN checking lock on 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:22,036 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=26 updating hbase:meta row=13dce85b09b0d1a075f661c037edf834, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:22,040 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=31, ppid=26, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=25 2018-10-08 03:44:22,176 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.UnassignRegionHandler(102): Close 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:22,176 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(102): Close 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:22,177 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1554): Closing 7beb91c361da5530dbf69d1615f54801, disabling compactions & flushes 2018-10-08 03:44:22,176 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(102): Close 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:22,178 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:22,178 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1554): Closing 7b2f6b48e0a7b7e4802db18892a95ae7, disabling compactions & flushes 2018-10-08 03:44:22,177 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 13dce85b09b0d1a075f661c037edf834, disabling compactions & flushes 2018-10-08 03:44:22,178 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:22,179 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:22,190 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:22,190 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:22,191 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2018-10-08 03:44:22,194 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1711): Closed testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. 2018-10-08 03:44:22,194 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. 2018-10-08 03:44:22,194 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1711): Closed testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. 2018-10-08 03:44:22,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583] assignment.RegionStateStore(200): pid=26 updating hbase:meta row=13dce85b09b0d1a075f661c037edf834, regionState=CLOSED 2018-10-08 03:44:22,196 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=28 updating hbase:meta row=7beb91c361da5530dbf69d1615f54801, regionState=CLOSED 2018-10-08 03:44:22,196 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=38583] assignment.RegionStateStore(200): pid=27 updating hbase:meta row=7b2f6b48e0a7b7e4802db18892a95ae7, regionState=CLOSED 2018-10-08 03:44:22,203 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(124): Closed 13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:22,203 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.UnassignRegionHandler(124): Closed 7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:22,203 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.UnassignRegionHandler(124): Closed 7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:22,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, UNASSIGN; resume parent processing. 2018-10-08 03:44:22,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=31, ppid=26, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 164msec 2018-10-08 03:44:22,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=26, ppid=25, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=13dce85b09b0d1a075f661c037edf834, UNASSIGN in 199msec 2018-10-08 03:44:22,218 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=27, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, UNASSIGN; resume parent processing. 2018-10-08 03:44:22,218 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=30, ppid=27, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 185msec 2018-10-08 03:44:22,221 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=27, ppid=25, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7b2f6b48e0a7b7e4802db18892a95ae7, UNASSIGN in 208msec 2018-10-08 03:44:22,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=28, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, UNASSIGN; resume parent processing. 2018-10-08 03:44:22,227 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=29, ppid=28, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 204msec 2018-10-08 03:44:22,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=25, state=RUNNABLE:DISABLE_TABLE_ADD_REPLICATION_BARRIER, hasLock=true; DisableTableProcedure table=testMergeWithoutPONR; resume parent processing. 2018-10-08 03:44:22,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=28, ppid=25, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testMergeWithoutPONR, region=7beb91c361da5530dbf69d1615f54801, UNASSIGN in 218msec 2018-10-08 03:44:22,234 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970262234}]},"ts":1538970262234} 2018-10-08 03:44:22,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testMergeWithoutPONR, state=DISABLED in hbase:meta 2018-10-08 03:44:22,240 INFO [PEWorker-1] procedure.DisableTableProcedure(307): Set testMergeWithoutPONR to state=DISABLED 2018-10-08 03:44:22,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR in 260msec 2018-10-08 03:44:22,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=25 2018-10-08 03:44:22,309 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3721): Operation: DISABLE, Table Name: default:testMergeWithoutPONR, procId: 25 completed 2018-10-08 03:44:22,317 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.HMaster$4(2366): Client=jenkins//67.195.81.148 delete testMergeWithoutPONR 2018-10-08 03:44:22,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] procedure2.ProcedureExecutor(1124): Stored pid=32, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,334 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(97): Waiting for RIT for pid=32, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,338 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(108): Deleting regions from META for pid=32, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262338}]},"ts":9223372036854775807} 2018-10-08 03:44:22,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262338}]},"ts":9223372036854775807} 2018-10-08 03:44:22,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262338}]},"ts":9223372036854775807} 2018-10-08 03:44:22,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=32 2018-10-08 03:44:22,345 INFO [PEWorker-1] hbase.MetaTableAccessor(1868): Deleted 3 regions from META 2018-10-08 03:44:22,345 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1869): Deleted regions: [{ENCODED => 13dce85b09b0d1a075f661c037edf834, NAME => 'testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7b2f6b48e0a7b7e4802db18892a95ae7, NAME => 'testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7beb91c361da5530dbf69d1615f54801, NAME => 'testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.', STARTKEY => '2', ENDKEY => ''}] 2018-10-08 03:44:22,354 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=32, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,359 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834. from FS 2018-10-08 03:44:22,359 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,368 DEBUG [PEWorker-1] backup.HFileArchiver(152): Archiving [FileablePath, hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834/FAMILY] 2018-10-08 03:44:22,376 INFO [IPC Server handler 9 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741842_1018 127.0.0.1:41173 2018-10-08 03:44:22,377 DEBUG [PEWorker-1] backup.HFileArchiver(458): Deleted hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/13dce85b09b0d1a075f661c037edf834 2018-10-08 03:44:22,378 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7. from FS 2018-10-08 03:44:22,378 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,382 DEBUG [PEWorker-1] backup.HFileArchiver(152): Archiving [FileablePath, hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7/FAMILY] 2018-10-08 03:44:22,388 INFO [IPC Server handler 1 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:41173 2018-10-08 03:44:22,388 DEBUG [PEWorker-1] backup.HFileArchiver(458): Deleted hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7b2f6b48e0a7b7e4802db18892a95ae7 2018-10-08 03:44:22,389 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801. from FS 2018-10-08 03:44:22,389 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,395 DEBUG [PEWorker-1] backup.HFileArchiver(152): Archiving [FileablePath, hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801/FAMILY] 2018-10-08 03:44:22,407 INFO [IPC Server handler 8 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741835_1011 127.0.0.1:41173 2018-10-08 03:44:22,408 DEBUG [PEWorker-1] backup.HFileArchiver(458): Deleted hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testMergeWithoutPONR/7beb91c361da5530dbf69d1615f54801 2018-10-08 03:44:22,408 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'testMergeWithoutPONR' archived! 2018-10-08 03:44:22,410 INFO [IPC Server handler 9 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741834_1010 127.0.0.1:41173 2018-10-08 03:44:22,411 INFO [IPC Server handler 9 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:41173 2018-10-08 03:44:22,411 INFO [IPC Server handler 9 on 54583] blockmanagement.BlockManager(1168): BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:41173 2018-10-08 03:44:22,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(119): Deleting descriptor for pid=32, state=RUNNABLE:DELETE_TABLE_UPDATE_DESC_CACHE, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(403): Removing 'testMergeWithoutPONR' descriptor. 2018-10-08 03:44:22,416 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=32, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,417 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(393): Removing 'testMergeWithoutPONR' from region states. 2018-10-08 03:44:22,417 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262417}]},"ts":9223372036854775807} 2018-10-08 03:44:22,417 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262417}]},"ts":9223372036854775807} 2018-10-08 03:44:22,418 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1538970262417}]},"ts":9223372036854775807} 2018-10-08 03:44:22,424 INFO [PEWorker-1] hbase.MetaTableAccessor(1868): Deleted 3 regions from META 2018-10-08 03:44:22,424 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1869): Deleted regions: [{ENCODED => 13dce85b09b0d1a075f661c037edf834, NAME => 'testMergeWithoutPONR,,1538970258406.13dce85b09b0d1a075f661c037edf834.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 7b2f6b48e0a7b7e4802db18892a95ae7, NAME => 'testMergeWithoutPONR,1,1538970255933.7b2f6b48e0a7b7e4802db18892a95ae7.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 7beb91c361da5530dbf69d1615f54801, NAME => 'testMergeWithoutPONR,2,1538970255933.7beb91c361da5530dbf69d1615f54801.', STARTKEY => '2', ENDKEY => ''}] 2018-10-08 03:44:22,424 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(397): Marking 'testMergeWithoutPONR' as deleted. 2018-10-08 03:44:22,425 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Delete {"totalColumns":1,"row":"testMergeWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1538970262425}]},"ts":9223372036854775807} 2018-10-08 03:44:22,429 INFO [PEWorker-1] hbase.MetaTableAccessor(1724): Deleted table testMergeWithoutPONR state from META 2018-10-08 03:44:22,432 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=32, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, hasLock=true; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:22,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR in 113msec 2018-10-08 03:44:22,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=32 2018-10-08 03:44:22,445 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3721): Operation: DELETE, Table Name: default:testMergeWithoutPONR, procId: 32 completed 2018-10-08 03:44:22,481 INFO [Time-limited test] hbase.ResourceChecker(172): after: master.assignment.TestMergeTableRegionsProcedure#testMergeWithoutPONR Thread=211 (was 197) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-921593623_23 at /127.0.0.1:45614 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf904:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf904:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1232054963-67.195.81.148-1538970244597:blk_1073741841_1017, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataStreamer for file /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log block BP-1232054963-67.195.81.148-1538970244597:blk_1073741841_1017 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:523) Potentially hanging thread: RSProcedureDispatcher-pool3-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-regionserver/asf904:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ResponseProcessor for block BP-1232054963-67.195.81.148-1538970244597:blk_1073741841_1017 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2292) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:847) Potentially hanging thread: RSProcedureDispatcher-pool3-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-602425802_23 at /127.0.0.1:45658 [Receiving block BP-1232054963-67.195.81.148-1538970244597:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:808) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-regionserver/asf904:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e502c51-shared-pool6-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5e7e1686-shared-pool10-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-602425802_23 at /127.0.0.1:45566 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=864 (was 848) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=618 (was 602) - SystemLoadAverage LEAK? -, ProcessCount=290 (was 289) - ProcessCount LEAK? -, AvailableMemoryMB=10198 (was 10675) 2018-10-08 03:44:22,510 INFO [Time-limited test] hbase.ResourceChecker(148): before: master.assignment.TestMergeTableRegionsProcedure#testRollbackAndDoubleExecution Thread=211, OpenFileDescriptor=864, MaxFileDescriptor=60000, SystemLoadAverage=618, ProcessCount=290, AvailableMemoryMB=10197 2018-10-08 03:44:22,510 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:22,515 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2018-10-08 03:44:22,517 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(422): Client=jenkins//67.195.81.148 set balanceSwitch=false 2018-10-08 03:44:22,517 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: false 2018-10-08 03:44:22,521 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.HMaster$3(2004): Client=jenkins//67.195.81.148 create 'testRollbackAndDoubleExecution', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-10-08 03:44:22,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] procedure2.ProcedureExecutor(1124): Stored pid=33, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:22,529 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(630): Client=jenkins//67.195.81.148 procedure request for creating table: namespace: "default" qualifier: "testRollbackAndDoubleExecution" procId is: 33 2018-10-08 03:44:22,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:22,532 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. from FS 2018-10-08 03:44:22,533 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,534 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb empty. 2018-10-08 03:44:22,535 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:22,536 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. from FS 2018-10-08 03:44:22,536 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,537 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00 empty. 2018-10-08 03:44:22,537 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:22,538 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. from FS 2018-10-08 03:44:22,538 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,539 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87 empty. 2018-10-08 03:44:22,540 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:22,540 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(320): Archiving region testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. from FS 2018-10-08 03:44:22,540 DEBUG [PEWorker-1] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904 2018-10-08 03:44:22,541 DEBUG [PEWorker-1] backup.HFileArchiver(146): Directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8 empty. 2018-10-08 03:44:22,542 DEBUG [PEWorker-1] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:22,542 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(324): Table 'testRollbackAndDoubleExecution' archived! 2018-10-08 03:44:22,558 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741843_1019{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 502 2018-10-08 03:44:22,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:22,963 DEBUG [PEWorker-1] util.FSTableDescriptors(683): Wrote into hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2018-10-08 03:44:22,968 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-4] regionserver.HRegion(7043): creating HRegion testRollbackAndDoubleExecution HTD == 'testRollbackAndDoubleExecution', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testRollbackAndDoubleExecution 2018-10-08 03:44:22,969 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-3] regionserver.HRegion(7043): creating HRegion testRollbackAndDoubleExecution HTD == 'testRollbackAndDoubleExecution', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testRollbackAndDoubleExecution 2018-10-08 03:44:22,970 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-2] regionserver.HRegion(7043): creating HRegion testRollbackAndDoubleExecution HTD == 'testRollbackAndDoubleExecution', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testRollbackAndDoubleExecution 2018-10-08 03:44:22,971 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-1] regionserver.HRegion(7043): creating HRegion testRollbackAndDoubleExecution HTD == 'testRollbackAndDoubleExecution', {NAME => 'FAMILY', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/.tmp Table name == testRollbackAndDoubleExecution 2018-10-08 03:44:23,009 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741844_1020{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 67 2018-10-08 03:44:23,011 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741845_1021{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 66 2018-10-08 03:44:23,011 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741847_1023{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 67 2018-10-08 03:44:23,011 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741846_1022{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 66 2018-10-08 03:44:23,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:23,411 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-3] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,412 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-3] regionserver.HRegion(1554): Closing 381466ba5ddfd0f7f6d889dbd8abea87, disabling compactions & flushes 2018-10-08 03:44:23,412 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-3] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,412 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-3] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,412 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-1] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,412 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-1] regionserver.HRegion(1554): Closing 9bac7c539ac0cff6dc5706ed375a3bfb, disabling compactions & flushes 2018-10-08 03:44:23,413 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-1] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,413 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-1] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,413 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-4] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,413 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-4] regionserver.HRegion(1554): Closing 8f69cb83fe1220eb0f370056e2b218e8, disabling compactions & flushes 2018-10-08 03:44:23,413 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-2] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,414 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-2] regionserver.HRegion(1554): Closing 2fa19437170a1cb8cf8dca0aac02ba00, disabling compactions & flushes 2018-10-08 03:44:23,414 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-2] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,414 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-2] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,414 DEBUG [RegionOpenAndInitThread-testRollbackAndDoubleExecution-4] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,415 INFO [RegionOpenAndInitThread-testRollbackAndDoubleExecution-4] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,423 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":1538970263422},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970263422}]},"ts":1538970263422} 2018-10-08 03:44:23,423 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":1538970263422},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970263422}]},"ts":1538970263422} 2018-10-08 03:44:23,424 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":1538970263422},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970263422}]},"ts":1538970263422} 2018-10-08 03:44:23,424 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":1538970263422},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1538970263422}]},"ts":1538970263422} 2018-10-08 03:44:23,431 INFO [PEWorker-1] hbase.MetaTableAccessor(1555): Added 4 regions to meta. 2018-10-08 03:44:23,433 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970263433}]},"ts":1538970263433} 2018-10-08 03:44:23,437 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2018-10-08 03:44:23,441 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=34, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, ASSIGN}, {pid=35, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, ASSIGN}, {pid=36, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=381466ba5ddfd0f7f6d889dbd8abea87, ASSIGN}, {pid=37, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=8f69cb83fe1220eb0f370056e2b218e8, ASSIGN}] 2018-10-08 03:44:23,446 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=37, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=8f69cb83fe1220eb0f370056e2b218e8, ASSIGN checking lock on 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,448 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=37, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=8f69cb83fe1220eb0f370056e2b218e8, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:23,450 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=36, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=381466ba5ddfd0f7f6d889dbd8abea87, ASSIGN checking lock on 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,451 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=36, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=381466ba5ddfd0f7f6d889dbd8abea87, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:23,453 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=35, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, ASSIGN checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,455 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=35, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:23,457 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=34, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, ASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,458 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(160): Starting pid=34, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, ASSIGN; rit=OFFLINE, location=asf904.gq1.ygridcore.net,47093,1538970250889; forceNewPlan=false, retain=false 2018-10-08 03:44:23,601 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=34 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=38, ppid=34, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:23,612 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=36 updating hbase:meta row=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:23,624 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=35 updating hbase:meta row=2fa19437170a1cb8cf8dca0aac02ba00, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=40, ppid=35, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:23,633 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=37 updating hbase:meta row=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPENING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=41, ppid=37, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2018-10-08 03:44:23,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:23,767 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(101): Open testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,768 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7217): Opening region: {ENCODED => 8f69cb83fe1220eb0f370056e2b218e8, NAME => 'testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8.', STARTKEY => '2', ENDKEY => ''} 2018-10-08 03:44:23,769 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,769 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,769 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7256): checking encryption for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,769 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(7261): checking classloading for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,772 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(101): Open testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,773 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7217): Opening region: {ENCODED => 9bac7c539ac0cff6dc5706ed375a3bfb, NAME => 'testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb.', STARTKEY => '', ENDKEY => '0'} 2018-10-08 03:44:23,773 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,773 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,774 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7256): checking encryption for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,774 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7261): checking classloading for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,777 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(101): Open testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,777 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7217): Opening region: {ENCODED => 2fa19437170a1cb8cf8dca0aac02ba00, NAME => 'testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00.', STARTKEY => '0', ENDKEY => '1'} 2018-10-08 03:44:23,778 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,778 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,778 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7256): checking encryption for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,778 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(7261): checking classloading for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,779 DEBUG [StoreOpener-8f69cb83fe1220eb0f370056e2b218e8-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8/FAMILY 2018-10-08 03:44:23,779 DEBUG [StoreOpener-8f69cb83fe1220eb0f370056e2b218e8-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8/FAMILY 2018-10-08 03:44:23,780 INFO [StoreOpener-8f69cb83fe1220eb0f370056e2b218e8-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:23,780 INFO [StoreOpener-8f69cb83fe1220eb0f370056e2b218e8-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:23,781 DEBUG [StoreOpener-9bac7c539ac0cff6dc5706ed375a3bfb-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb/FAMILY 2018-10-08 03:44:23,782 DEBUG [StoreOpener-9bac7c539ac0cff6dc5706ed375a3bfb-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb/FAMILY 2018-10-08 03:44:23,783 INFO [StoreOpener-8f69cb83fe1220eb0f370056e2b218e8-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:23,783 INFO [StoreOpener-9bac7c539ac0cff6dc5706ed375a3bfb-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:23,784 INFO [StoreOpener-9bac7c539ac0cff6dc5706ed375a3bfb-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:23,785 INFO [StoreOpener-9bac7c539ac0cff6dc5706ed375a3bfb-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:23,785 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(949): replaying wal for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,786 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(949): replaying wal for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,790 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,790 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,792 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,792 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(957): stopping wal replay for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,792 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(969): Cleaning up temporary data for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,792 DEBUG [StoreOpener-2fa19437170a1cb8cf8dca0aac02ba00-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00/FAMILY 2018-10-08 03:44:23,792 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,792 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(957): stopping wal replay for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,793 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(969): Cleaning up temporary data for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,792 DEBUG [StoreOpener-2fa19437170a1cb8cf8dca0aac02ba00-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00/FAMILY 2018-10-08 03:44:23,794 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(980): Cleaning up detritus for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,794 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(980): Cleaning up detritus for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,795 INFO [StoreOpener-2fa19437170a1cb8cf8dca0aac02ba00-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:23,796 INFO [StoreOpener-2fa19437170a1cb8cf8dca0aac02ba00-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:23,796 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1005): writing seq id for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,797 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1005): writing seq id for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,806 INFO [StoreOpener-2fa19437170a1cb8cf8dca0aac02ba00-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:23,806 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(949): replaying wal for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,810 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,811 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,811 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(957): stopping wal replay for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,811 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(969): Cleaning up temporary data for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,811 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:23,811 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:23,812 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1009): Opened 9bac7c539ac0cff6dc5706ed375a3bfb; next sequenceid=2 2018-10-08 03:44:23,812 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1016): Running coprocessor post-open hooks for 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:23,812 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1009): Opened 8f69cb83fe1220eb0f370056e2b218e8; next sequenceid=2 2018-10-08 03:44:23,812 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegion(1016): Running coprocessor post-open hooks for 8f69cb83fe1220eb0f370056e2b218e8 2018-10-08 03:44:23,813 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(980): Cleaning up detritus for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,814 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2198): Post open deploy tasks for testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,814 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2198): Post open deploy tasks for testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,816 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1005): writing seq id for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,822 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=37 updating hbase:meta row=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,823 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=38583] assignment.RegionStateStore(200): pid=34 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,826 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:23,826 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1009): Opened 2fa19437170a1cb8cf8dca0aac02ba00; next sequenceid=2 2018-10-08 03:44:23,826 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegion(1016): Running coprocessor post-open hooks for 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:23,827 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2222): Finished post open deploy task for testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,827 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(138): Opened testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:23,827 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(101): Open testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,827 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-2] regionserver.HRegionServer(2222): Finished post open deploy task for testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,828 INFO [RS_OPEN_REGION-regionserver/asf904:0-2] handler.AssignRegionHandler(138): Opened testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:44:23,828 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7217): Opening region: {ENCODED => 381466ba5ddfd0f7f6d889dbd8abea87, NAME => 'testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87.', STARTKEY => '1', ENDKEY => '2'} 2018-10-08 03:44:23,828 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,829 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(836): Instantiated testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-10-08 03:44:23,829 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7256): checking encryption for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,829 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(7261): checking classloading for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,833 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2198): Post open deploy tasks for testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,835 DEBUG [StoreOpener-381466ba5ddfd0f7f6d889dbd8abea87-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87/FAMILY 2018-10-08 03:44:23,835 DEBUG [StoreOpener-381466ba5ddfd0f7f6d889dbd8abea87-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87/FAMILY 2018-10-08 03:44:23,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=34, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, ASSIGN; resume parent processing. 2018-10-08 03:44:23,835 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=35 updating hbase:meta row=2fa19437170a1cb8cf8dca0aac02ba00, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=38, ppid=34, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 225msec 2018-10-08 03:44:23,837 INFO [StoreOpener-381466ba5ddfd0f7f6d889dbd8abea87-1] hfile.CacheConfig(239): Created cacheConfig for FAMILY: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-10-08 03:44:23,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=34, ppid=33, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, ASSIGN in 395msec 2018-10-08 03:44:23,839 INFO [StoreOpener-381466ba5ddfd0f7f6d889dbd8abea87-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-10-08 03:44:23,842 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-0] regionserver.HRegionServer(2222): Finished post open deploy task for testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,843 INFO [StoreOpener-381466ba5ddfd0f7f6d889dbd8abea87-1] regionserver.HStore(327): Store=FAMILY, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-10-08 03:44:23,844 INFO [RS_OPEN_REGION-regionserver/asf904:0-0] handler.AssignRegionHandler(138): Opened testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:23,844 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(949): replaying wal for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=37, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=8f69cb83fe1220eb0f370056e2b218e8, ASSIGN; resume parent processing. 2018-10-08 03:44:23,847 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=41, ppid=37, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 205msec 2018-10-08 03:44:23,847 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(4611): Found 0 recovered edits file(s) under hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,848 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(957): stopping wal replay for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,848 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(969): Cleaning up temporary data for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,848 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(980): Cleaning up detritus for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=37, ppid=33, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=8f69cb83fe1220eb0f370056e2b218e8, ASSIGN in 406msec 2018-10-08 03:44:23,851 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1005): writing seq id for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,860 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-10-08 03:44:23,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=35, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, ASSIGN; resume parent processing. 2018-10-08 03:44:23,861 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1009): Opened 381466ba5ddfd0f7f6d889dbd8abea87; next sequenceid=2 2018-10-08 03:44:23,861 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegion(1016): Running coprocessor post-open hooks for 381466ba5ddfd0f7f6d889dbd8abea87 2018-10-08 03:44:23,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=40, ppid=35, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 223msec 2018-10-08 03:44:23,863 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2198): Post open deploy tasks for testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,864 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=35, ppid=33, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, ASSIGN in 420msec 2018-10-08 03:44:23,867 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=36 updating hbase:meta row=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, openSeqNum=2, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:23,873 DEBUG [RS_OPEN_REGION-regionserver/asf904:0-1] regionserver.HRegionServer(2222): Finished post open deploy task for testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,874 INFO [RS_OPEN_REGION-regionserver/asf904:0-1] handler.AssignRegionHandler(138): Opened testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:44:23,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=36, ppid=33, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=381466ba5ddfd0f7f6d889dbd8abea87, ASSIGN; resume parent processing. 2018-10-08 03:44:23,876 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=39, ppid=36, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 256msec 2018-10-08 03:44:23,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=33, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution; resume parent processing. 2018-10-08 03:44:23,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=36, ppid=33, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=381466ba5ddfd0f7f6d889dbd8abea87, ASSIGN in 435msec 2018-10-08 03:44:23,881 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2180): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1538970263880}]},"ts":1538970263880} 2018-10-08 03:44:23,884 INFO [PEWorker-1] hbase.MetaTableAccessor(1700): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2018-10-08 03:44:23,890 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution in 1.3670sec 2018-10-08 03:44:24,305 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1b130f2f] blockmanagement.BlockManager(3480): BLOCK* BlockManager: ask 127.0.0.1:41173 to delete [blk_1073741842_1018, blk_1073741834_1010, blk_1073741835_1011, blk_1073741836_1012, blk_1073741837_1013, blk_1073741838_1014] 2018-10-08 03:44:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] master.MasterRpcServices(1175): Checking to see if procedure is done pid=33 2018-10-08 03:44:24,648 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3721): Operation: CREATE, Table Name: default:testRollbackAndDoubleExecution, procId: 33 completed 2018-10-08 03:44:24,649 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [900,000] milli-secs(wait.for.ratio=[1]) 2018-10-08 03:44:24,906 WARN [Time-limited test] procedure2.ProcedureTestingUtility(163): Set Kill before store update to: true 2018-10-08 03:44:24,912 DEBUG [Time-limited test] procedure2.ProcedureExecutor(1124): Stored pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true 2018-10-08 03:44:24,913 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:24,913 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:24,913 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:24,935 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:24,935 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true 2018-10-08 03:44:24,935 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:24,935 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:25,162 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(404): Restart 3 exec state=pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=true 2018-10-08 03:44:25,162 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:25,162 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:25,168 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 1107 2018-10-08 03:44:25,169 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:25,169 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:25,170 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:25,171 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:25,173 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 2ms 2018-10-08 03:44:25,173 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,179 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:25,180 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:25,180 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,187 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:25,189 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:25,189 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,196 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:25,197 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:25,197 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,205 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=5 2018-10-08 03:44:25,207 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 37msec 2018-10-08 03:44:25,216 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:25,220 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:25,223 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:25,228 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:25,230 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:25,230 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,231 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,232 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:25,233 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,233 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:25,234 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:25,235 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:25,236 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:25,236 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,236 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:25,237 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:25,237 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:25,237 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:25,238 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 30msec 2018-10-08 03:44:25,240 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970295239 2018-10-08 03:44:25,241 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PREPARE, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,242 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:25,252 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,254 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,254 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,256 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,257 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,257 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:25,257 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970325257 2018-10-08 03:44:25,258 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 16msec 2018-10-08 03:44:25,271 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:25,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:25,282 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:25,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:25,508 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(404): Restart 4 exec state=pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,508 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:25,509 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:25,514 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741848_1024{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 504 2018-10-08 03:44:25,915 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:25,917 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:25,917 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:25,920 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:25,920 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 0ms 2018-10-08 03:44:25,921 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,926 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:25,927 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:25,927 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,932 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:25,933 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 0ms 2018-10-08 03:44:25,934 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,941 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:25,942 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:25,942 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,947 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:25,948 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 0ms 2018-10-08 03:44:25,949 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:25,956 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=6 2018-10-08 03:44:25,958 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 40msec 2018-10-08 03:44:25,963 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:25,968 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:25,971 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:25,974 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:25,982 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:25,983 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:25,983 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,984 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,984 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:25,985 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:25,985 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:25,985 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:25,986 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:25,986 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:25,986 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,987 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:25,987 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:25,987 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:25,987 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:25,988 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 28msec 2018-10-08 03:44:25,990 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970295990 2018-10-08 03:44:25,991 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:25,991 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_PRE_MERGE_OPERATION, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:25,999 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:25,999 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,000 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,001 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,001 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,002 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:26,002 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970326002 2018-10-08 03:44:26,002 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 11msec 2018-10-08 03:44:26,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN}, {pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN}] 2018-10-08 03:44:26,019 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:26,021 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:26,023 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=44 updating hbase:meta row=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:26,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:26,027 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:26,027 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:26,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:26,028 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:26,255 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 0 rollback state: pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:26,256 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:26,256 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:26,263 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741849_1025{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 2008 2018-10-08 03:44:26,404 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-08 03:44:26,664 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:26,665 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:26,666 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:26,668 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:26,669 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-10-08 03:44:26,669 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,676 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:26,677 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:26,678 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,683 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:26,684 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:26,684 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,689 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:26,690 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:26,690 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,696 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:26,697 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-08 03:44:26,697 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,701 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:26,702 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 1ms 2018-10-08 03:44:26,702 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:26,708 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=7 2018-10-08 03:44:26,710 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 43msec 2018-10-08 03:44:26,713 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:26,719 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:26,724 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:26,727 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:26,730 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:26,736 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:26,740 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:26,740 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:26,741 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:26,742 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:26,743 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:26,743 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:26,744 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:26,744 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:26,745 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:26,745 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:26,745 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:26,745 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:26,745 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:26,746 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:26,746 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:26,746 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:26,746 DEBUG [Time-limited test] procedure2.Procedure(965): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:26,747 DEBUG [Time-limited test] procedure2.Procedure(979): pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:26,747 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:26,747 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 37msec 2018-10-08 03:44:26,748 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb to restore RIT 2018-10-08 03:44:26,749 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00 to restore RIT 2018-10-08 03:44:26,749 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970296749 2018-10-08 03:44:26,749 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:26,751 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:26,752 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:26,757 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,758 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,758 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,759 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,759 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:26,760 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:26,760 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970326760 2018-10-08 03:44:26,760 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=44 updating hbase:meta row=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:26,760 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 10msec 2018-10-08 03:44:26,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:26,766 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:26,783 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:26,785 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=43 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:26,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=46, ppid=43, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:26,792 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:26,793 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:26,793 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:26,793 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:26,933 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(102): Close 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:26,934 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 2fa19437170a1cb8cf8dca0aac02ba00, disabling compactions & flushes 2018-10-08 03:44:26,934 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:26,942 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/2fa19437170a1cb8cf8dca0aac02ba00/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:26,944 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,0,1538970262521.2fa19437170a1cb8cf8dca0aac02ba00. 2018-10-08 03:44:26,945 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=44 updating hbase:meta row=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED 2018-10-08 03:44:26,950 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.UnassignRegionHandler(124): Closed 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:27,011 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 1 rollback state: pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:27,011 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:27,011 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:27,016 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741850_1026{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 2142 2018-10-08 03:44:27,417 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:27,418 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:27,419 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:27,420 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:27,421 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-10-08 03:44:27,421 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,426 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:27,427 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:27,428 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,432 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:27,434 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:27,434 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,439 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:27,440 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:27,440 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,444 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:27,445 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 0ms 2018-10-08 03:44:27,445 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,450 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:27,451 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 0ms 2018-10-08 03:44:27,451 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,455 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:27,456 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 1ms 2018-10-08 03:44:27,457 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:27,474 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=8 2018-10-08 03:44:27,476 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 56msec 2018-10-08 03:44:27,479 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:27,490 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:27,494 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:27,499 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:27,502 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:27,506 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:27,512 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:27,517 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:27,517 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:27,517 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:27,523 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:27,525 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:27,525 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:27,525 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:27,527 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:27,527 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:27,527 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:27,527 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:27,527 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:27,528 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:27,528 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:27,528 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:27,528 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:27,528 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:27,529 DEBUG [Time-limited test] procedure2.Procedure(979): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:27,529 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:27,529 DEBUG [Time-limited test] procedure2.Procedure(979): pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:27,529 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:27,529 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:27,530 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 53msec 2018-10-08 03:44:27,531 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb to restore RIT 2018-10-08 03:44:27,531 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=44, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00 to restore RIT 2018-10-08 03:44:27,531 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970297531 2018-10-08 03:44:27,532 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:27,533 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:27,542 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:27,543 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:27,544 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:27,544 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:27,545 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:27,545 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:27,545 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970327545 2018-10-08 03:44:27,545 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 13msec 2018-10-08 03:44:27,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=44, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN; resume parent processing. 2018-10-08 03:44:27,548 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 779msec 2018-10-08 03:44:27,548 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:27,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN in 1.5310sec 2018-10-08 03:44:27,551 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=43 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:27,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=46, ppid=43, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:27,555 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:27,555 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:27,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:27,555 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:27,796 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 2 rollback state: pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:27,796 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:27,796 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:27,801 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741851_1027{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 2188 2018-10-08 03:44:28,203 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:28,204 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:28,205 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:28,207 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:28,208 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-10-08 03:44:28,208 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,213 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:28,214 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:28,214 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,218 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:28,219 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 0ms 2018-10-08 03:44:28,219 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,223 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:28,224 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:28,224 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,230 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:28,231 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-08 03:44:28,231 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,235 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:28,236 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 0ms 2018-10-08 03:44:28,237 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,240 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:28,241 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 0ms 2018-10-08 03:44:28,241 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,245 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:28,246 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log after 1ms 2018-10-08 03:44:28,246 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2188; replication=1; blocksize=134217728; modification_time=1538970268203; access_time=1538970267472; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,253 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=9 2018-10-08 03:44:28,255 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 49msec 2018-10-08 03:44:28,258 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:28,261 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:28,264 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:28,266 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:28,270 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:28,272 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:28,274 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:28,278 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:28,280 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:28,280 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:28,281 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:28,281 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:28,282 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:28,282 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:28,283 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:28,283 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:28,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:28,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:28,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:28,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:28,285 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:28,285 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:28,285 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:28,285 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:28,285 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:28,286 DEBUG [Time-limited test] procedure2.Procedure(979): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:28,286 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:28,286 DEBUG [Time-limited test] procedure2.Procedure(965): pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:28,286 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:28,287 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 31msec 2018-10-08 03:44:28,288 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb to restore RIT 2018-10-08 03:44:28,288 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970298288 2018-10-08 03:44:28,288 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:28,290 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:28,296 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:28,297 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSING, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:28,297 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:28,297 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:28,298 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:28,298 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:28,298 INFO [PEWorker-1] assignment.RegionStateStore(200): pid=43 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSING, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:44:28,298 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970328298 2018-10-08 03:44:28,299 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 10msec 2018-10-08 03:44:28,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1738): Initialized subprocedures=[{pid=46, ppid=43, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2018-10-08 03:44:28,303 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:28,467 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.UnassignRegionHandler(102): Close 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:28,468 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1554): Closing 9bac7c539ac0cff6dc5706ed375a3bfb, disabling compactions & flushes 2018-10-08 03:44:28,468 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:28,478 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/9bac7c539ac0cff6dc5706ed375a3bfb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:44:28,480 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,,1538970262521.9bac7c539ac0cff6dc5706ed375a3bfb. 2018-10-08 03:44:28,482 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583] assignment.RegionStateStore(200): pid=43 updating hbase:meta row=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSED 2018-10-08 03:44:28,487 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.UnassignRegionHandler(124): Closed 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:28,488 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:28,488 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=46, ppid=43, state=SUCCESS, hasLock=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:28,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:28,489 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=43, state=SUCCESS, hasLock=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:28,549 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 3 rollback state: pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:28,549 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:28,549 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:28,555 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741852_1028{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 2133 2018-10-08 03:44:28,957 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:28,958 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:28,958 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:28,960 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:28,962 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 2ms 2018-10-08 03:44:28,962 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,980 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:28,982 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 2ms 2018-10-08 03:44:28,982 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,993 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:28,994 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:28,994 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:28,998 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:29,012 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 14ms 2018-10-08 03:44:29,012 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,026 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:29,027 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-08 03:44:29,027 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,036 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:29,037 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 0ms 2018-10-08 03:44:29,038 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,042 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:29,044 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 2ms 2018-10-08 03:44:29,045 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,048 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:29,049 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log after 0ms 2018-10-08 03:44:29,049 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2188; replication=1; blocksize=134217728; modification_time=1538970268203; access_time=1538970267472; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,054 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:29,055 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log after 0ms 2018-10-08 03:44:29,056 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log; isDirectory=false; length=2133; replication=1; blocksize=134217728; modification_time=1538970268956; access_time=1538970268252; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,069 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=10 2018-10-08 03:44:29,075 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 116msec 2018-10-08 03:44:29,079 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:29,082 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:29,084 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:29,088 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:29,091 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:29,095 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:29,102 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:29,106 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:29,110 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:29,113 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:29,113 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,114 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,114 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:29,116 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,117 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:29,117 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:29,117 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:29,117 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:29,118 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=46, ppid=43, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:29,118 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:29,118 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:29,118 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:29,118 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,119 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:29,119 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:29,119 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:29,119 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=WAITING:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:29,119 DEBUG [Time-limited test] procedure2.Procedure(979): pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:29,120 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:29,120 DEBUG [Time-limited test] procedure2.Procedure(965): pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,120 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,120 DEBUG [Time-limited test] procedure2.Procedure(979): pid=46, ppid=43, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:29,120 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 45msec 2018-10-08 03:44:29,122 INFO [Time-limited test] assignment.AssignmentManager(251): Attach pid=43, ppid=42, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN to rit=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb to restore RIT 2018-10-08 03:44:29,122 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970299122 2018-10-08 03:44:29,122 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:29,131 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,132 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,133 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,133 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,133 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,134 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:29,134 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970329134 2018-10-08 03:44:29,134 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 11msec 2018-10-08 03:44:29,136 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=43, ppid=42, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN; resume parent processing. 2018-10-08 03:44:29,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 820msec 2018-10-08 03:44:29,137 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:29,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1878): Finished subprocedure(s) of pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false; resume parent processing. 2018-10-08 03:44:29,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1507): Finished pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN in 3.1200sec 2018-10-08 03:44:29,154 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:29,154 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1813): TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CREATE_MERGED_REGION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,154 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:29,155 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CREATE_MERGED_REGION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1815) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1759) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1489) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:29,385 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 4 rollback state: pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CREATE_MERGED_REGION, hasLock=true; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,385 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:29,385 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:29,391 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 358 2018-10-08 03:44:29,394 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:29,395 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:29,395 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:29,397 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:29,398 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 1ms 2018-10-08 03:44:29,398 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,403 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:29,404 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 0ms 2018-10-08 03:44:29,405 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,409 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:29,410 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:29,410 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,414 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:29,415 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:29,415 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,419 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:29,420 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 0ms 2018-10-08 03:44:29,421 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,424 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:29,426 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 2ms 2018-10-08 03:44:29,426 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,430 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:29,431 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 0ms 2018-10-08 03:44:29,431 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,435 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:29,436 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log after 1ms 2018-10-08 03:44:29,436 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2188; replication=1; blocksize=134217728; modification_time=1538970268203; access_time=1538970267472; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,440 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:29,440 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log after 0ms 2018-10-08 03:44:29,440 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log; isDirectory=false; length=2133; replication=1; blocksize=134217728; modification_time=1538970268956; access_time=1538970268252; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,444 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:29,445 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log after 1ms 2018-10-08 03:44:29,445 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log; isDirectory=false; length=2291; replication=1; blocksize=134217728; modification_time=1538970269393; access_time=1538970269068; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:29,451 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=11 2018-10-08 03:44:29,452 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 56msec 2018-10-08 03:44:29,455 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:29,457 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:29,459 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:29,462 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:29,464 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:29,466 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:29,470 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:29,472 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:29,474 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:29,478 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:29,481 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:29,481 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,482 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,482 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:29,483 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:29,483 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:29,483 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:29,484 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.Procedure(965): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.Procedure(965): pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.Procedure(965): pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,485 DEBUG [Time-limited test] procedure2.Procedure(965): pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:29,486 DEBUG [Time-limited test] procedure2.StateMachineProcedure(221): Abort requested for pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,486 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 33msec 2018-10-08 03:44:29,488 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970299488 2018-10-08 03:44:29,488 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:29,489 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1497): LOCK_EVENT_WAIT pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,494 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,495 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,495 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,496 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,496 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:29,496 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:29,496 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970329496 2018-10-08 03:44:29,496 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 8msec 2018-10-08 03:44:29,497 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:29,497 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:29,497 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=42, state=RUNNABLE:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:29,508 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:29,525 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:29,525 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1621): TESTING: Kill before store update 2018-10-08 03:44:29,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:29,525 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1623) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1567) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1446) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:29,748 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 5 rollback state: pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:29,748 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:29,748 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:29,752 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741854_1030{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 1848 2018-10-08 03:44:30,156 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:30,157 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:30,157 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:30,159 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:30,160 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 0ms 2018-10-08 03:44:30,161 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,165 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:30,166 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:30,166 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,170 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:30,171 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:30,171 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,176 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:30,178 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 2ms 2018-10-08 03:44:30,179 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,184 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:30,185 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-08 03:44:30,185 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,189 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:30,190 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 1ms 2018-10-08 03:44:30,190 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,195 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:30,196 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 1ms 2018-10-08 03:44:30,196 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,202 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:30,203 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log after 1ms 2018-10-08 03:44:30,203 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2188; replication=1; blocksize=134217728; modification_time=1538970268203; access_time=1538970267472; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,207 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:30,208 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log after 1ms 2018-10-08 03:44:30,209 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log; isDirectory=false; length=2133; replication=1; blocksize=134217728; modification_time=1538970268956; access_time=1538970268252; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,213 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:30,214 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log after 1ms 2018-10-08 03:44:30,214 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log; isDirectory=false; length=2291; replication=1; blocksize=134217728; modification_time=1538970269393; access_time=1538970269068; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,219 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log 2018-10-08 03:44:30,220 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log after 1ms 2018-10-08 03:44:30,220 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log; isDirectory=false; length=1848; replication=1; blocksize=134217728; modification_time=1538970270155; access_time=1538970269450; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,229 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=12 2018-10-08 03:44:30,231 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 73msec 2018-10-08 03:44:30,240 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log 2018-10-08 03:44:30,243 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:30,246 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:30,248 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:30,251 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:30,262 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:30,265 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:30,269 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:30,274 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:30,277 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:30,279 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:30,283 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:30,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:30,284 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:30,285 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:30,286 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:30,286 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:30,286 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:30,286 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:30,287 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:30,288 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:30,288 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:30,288 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:30,288 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:30,288 DEBUG [Time-limited test] procedure2.Procedure(965): pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:30,289 DEBUG [Time-limited test] procedure2.Procedure(965): pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:30,289 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:30,289 DEBUG [Time-limited test] procedure2.Procedure(965): pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:30,289 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 57msec 2018-10-08 03:44:30,291 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970300291 2018-10-08 03:44:30,291 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:30,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1454): LOCK_EVENT_WAIT rollback...pid=42, state=FAILED:MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:30,302 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:30,302 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:30,303 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:30,303 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:30,304 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:30,304 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:30,304 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970330304 2018-10-08 03:44:30,304 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 12msec 2018-10-08 03:44:30,305 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:44:30,315 INFO [PEWorker-1] procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:30,320 ERROR [PEWorker-1] helpers.MarkerIgnoringBase(159): CODE-BUG: Uncaught runtime exception for pid=43, ppid=42, state=SUCCESS, hasLock=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN java.lang.UnsupportedOperationException at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:458) at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:97) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:208) at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:957) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1605) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1567) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1446) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:30,320 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: false 2018-10-08 03:44:30,320 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1621): TESTING: Kill before store update 2018-10-08 03:44:30,320 INFO [PEWorker-1] procedure2.ProcedureExecutor(705): Stopping 2018-10-08 03:44:30,321 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(2026): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1623) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1567) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1446) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2010) 2018-10-08 03:44:30,555 INFO [Time-limited test] procedure.MasterProcedureTestingUtility(497): Restart 6 rollback state: pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:30,555 INFO [Time-limited test] procedure2.ProcedureTestingUtility(94): RESTART - Stop 2018-10-08 03:44:30,555 INFO [Time-limited test] wal.WALProcedureStore(326): Stopping the WAL Procedure Store, isAbort=false 2018-10-08 03:44:30,560 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741855_1031{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 1752 2018-10-08 03:44:30,962 INFO [Time-limited test] assignment.AssignmentManager(261): Stopping assignment manager 2018-10-08 03:44:30,965 INFO [Time-limited test] procedure2.ProcedureTestingUtility(106): RESTART - Start 2018-10-08 03:44:30,966 INFO [Time-limited test] procedure2.ProcedureExecutor(641): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2018-10-08 03:44:30,971 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:30,974 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log after 2ms 2018-10-08 03:44:30,974 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log; isDirectory=false; length=56917; replication=1; blocksize=134217728; modification_time=1538970259096; access_time=1538970252557; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,981 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:30,982 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log after 1ms 2018-10-08 03:44:30,982 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log; isDirectory=false; length=470; replication=1; blocksize=134217728; modification_time=1538970259877; access_time=1538970259132; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,985 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:30,986 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log after 1ms 2018-10-08 03:44:30,986 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log; isDirectory=false; length=474; replication=1; blocksize=134217728; modification_time=1538970260612; access_time=1538970259901; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,989 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:30,990 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log after 1ms 2018-10-08 03:44:30,990 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log; isDirectory=false; length=72869; replication=1; blocksize=134217728; modification_time=1538970265168; access_time=1538970260647; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:30,996 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:30,997 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log after 1ms 2018-10-08 03:44:30,997 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log; isDirectory=false; length=504; replication=1; blocksize=134217728; modification_time=1538970265915; access_time=1538970265203; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,002 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:31,003 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log after 1ms 2018-10-08 03:44:31,003 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log; isDirectory=false; length=2008; replication=1; blocksize=134217728; modification_time=1538970266663; access_time=1538970265955; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,008 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:31,009 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log after 1ms 2018-10-08 03:44:31,009 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log; isDirectory=false; length=2142; replication=1; blocksize=134217728; modification_time=1538970267417; access_time=1538970266707; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,014 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:31,015 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log after 1ms 2018-10-08 03:44:31,015 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log; isDirectory=false; length=2188; replication=1; blocksize=134217728; modification_time=1538970268203; access_time=1538970267472; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,019 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:31,020 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log after 0ms 2018-10-08 03:44:31,021 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log; isDirectory=false; length=2133; replication=1; blocksize=134217728; modification_time=1538970268956; access_time=1538970268252; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,024 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:31,025 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log after 0ms 2018-10-08 03:44:31,026 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log; isDirectory=false; length=2291; replication=1; blocksize=134217728; modification_time=1538970269393; access_time=1538970269068; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,031 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log 2018-10-08 03:44:31,032 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log after 1ms 2018-10-08 03:44:31,032 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log; isDirectory=false; length=1848; replication=1; blocksize=134217728; modification_time=1538970270155; access_time=1538970269450; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,036 INFO [Time-limited test] util.FSHDFSUtils(199): Recover lease on dfs file hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000012.log 2018-10-08 03:44:31,037 INFO [Time-limited test] util.FSHDFSUtils(288): Recovered lease, attempt=0 on file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000012.log after 0ms 2018-10-08 03:44:31,037 DEBUG [Time-limited test] wal.WALProcedureStore(1327): Opening Pv2 FileStatus{path=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000012.log; isDirectory=false; length=1752; replication=1; blocksize=134217728; modification_time=1538970270961; access_time=1538970270228; owner=jenkins; group=supergroup; permission=rw-r--r--; isSymlink=false} 2018-10-08 03:44:31,043 INFO [Time-limited test] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=13 2018-10-08 03:44:31,045 INFO [Time-limited test] procedure2.ProcedureExecutor(660): Recovered WALProcedureStore lease in 79msec 2018-10-08 03:44:31,049 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000012.log 2018-10-08 03:44:31,051 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 3 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000011.log 2018-10-08 03:44:31,054 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000010.log 2018-10-08 03:44:31,057 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000009.log 2018-10-08 03:44:31,059 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 7 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000008.log 2018-10-08 03:44:31,061 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 6 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000007.log 2018-10-08 03:44:31,063 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 4 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000006.log 2018-10-08 03:44:31,065 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000005.log 2018-10-08 03:44:31,069 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 162 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000004.log 2018-10-08 03:44:31,072 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000003.log 2018-10-08 03:44:31,074 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 2 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000002.log 2018-10-08 03:44:31,077 INFO [Time-limited test] wal.ProcedureWALFormatReader(168): Read 123 entries in hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/MasterProcWALs/pv2-00000000000000000001.log 2018-10-08 03:44:31,080 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=33, state=SUCCESS, hasLock=false; CreateTableProcedure table=testRollbackAndDoubleExecution 2018-10-08 03:44:31,081 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=32, state=SUCCESS, hasLock=false; DeleteTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:31,081 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=25, state=SUCCESS, hasLock=false; DisableTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:31,082 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=18, state=SUCCESS, hasLock=false; MergeTableRegionsProcedure table=testMergeWithoutPONR, regions=[0a1f839303f18eb89f2dd2858e8f53e6, 64b6252f0678baac0ab16d1191421dec], forcibly=false 2018-10-08 03:44:31,083 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=9, state=SUCCESS, hasLock=false; CreateTableProcedure table=testMergeWithoutPONR 2018-10-08 03:44:31,083 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=8, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=hbase 2018-10-08 03:44:31,084 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=7, state=SUCCESS, hasLock=false; CreateNamespaceProcedure, namespace=default 2018-10-08 03:44:31,084 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=4, state=SUCCESS, hasLock=false; CreateTableProcedure table=hbase:namespace 2018-10-08 03:44:31,084 DEBUG [Time-limited test] procedure2.ProcedureExecutor(474): Completed pid=1, state=SUCCESS, hasLock=false; InitMetaProcedure table=hbase:meta 2018-10-08 03:44:31,084 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN 2018-10-08 03:44:31,084 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:31,085 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:31,085 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN 2018-10-08 03:44:31,085 DEBUG [Time-limited test] procedure2.ProcedureExecutor(523): Loading pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2018-10-08 03:44:31,087 DEBUG [Time-limited test] procedure2.Procedure(979): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false held the lock before restarting, call acquireLock to restore it. 2018-10-08 03:44:31,087 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 9bac7c539ac0cff6dc5706ed375a3bfb 2018-10-08 03:44:31,088 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 6f55ff84e5ac7395cb13c65de9fc62b9 2018-10-08 03:44:31,088 INFO [Time-limited test] procedure.MasterProcedureScheduler(689): pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false checking lock on 2fa19437170a1cb8cf8dca0aac02ba00 2018-10-08 03:44:31,088 DEBUG [Time-limited test] procedure2.Procedure(970): pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN is already finished, skip acquiring lock. 2018-10-08 03:44:31,088 DEBUG [Time-limited test] procedure2.Procedure(965): pid=44, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=2fa19437170a1cb8cf8dca0aac02ba00, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:31,088 DEBUG [Time-limited test] procedure2.Procedure(965): pid=45, ppid=44, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:31,089 DEBUG [Time-limited test] procedure2.Procedure(965): pid=46, ppid=43, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure didn't hold the lock before restarting, skip acquiring lock. 2018-10-08 03:44:31,089 INFO [Time-limited test] procedure2.ProcedureExecutor(674): Loaded WALProcedureStore in 43msec 2018-10-08 03:44:31,090 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1538970301090 2018-10-08 03:44:31,091 DEBUG [Time-limited test] assignment.AssignmentManager(1214): Joining cluster... 2018-10-08 03:44:31,092 INFO [PEWorker-1] procedure2.ProcedureExecutor(1454): LOCK_EVENT_WAIT rollback...pid=42, state=FAILED:MERGE_TABLE_REGIONS_CLOSE_REGIONS, hasLock=false, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via MergeTableRegionsProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; MergeTableRegionsProcedure table=testRollbackAndDoubleExecution, regions=[9bac7c539ac0cff6dc5706ed375a3bfb, 2fa19437170a1cb8cf8dca0aac02ba00], forcibly=false 2018-10-08 03:44:31,099 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=e298524ff1440c61e0c0af4e0880daf3, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:31,099 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=9bac7c539ac0cff6dc5706ed375a3bfb, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:31,100 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=2fa19437170a1cb8cf8dca0aac02ba00, regionState=CLOSED, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:31,100 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=381466ba5ddfd0f7f6d889dbd8abea87, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:31,100 INFO [Time-limited test] assignment.RegionStateStore(123): Load hbase:meta entry region=8f69cb83fe1220eb0f370056e2b218e8, regionState=OPEN, lastHost=asf904.gq1.ygridcore.net,47093,1538970250889, regionLocation=asf904.gq1.ygridcore.net,47093,1538970250889, openSeqNum=2 2018-10-08 03:44:31,101 INFO [Time-limited test] assignment.AssignmentManager(1226): Number of RegionServers=1 2018-10-08 03:44:31,101 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1425): pid=43, ppid=42, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN is already finished, skipping execution 2018-10-08 03:44:31,101 INFO [Time-limited test] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1538970331101 2018-10-08 03:44:31,101 INFO [Time-limited test] assignment.AssignmentManager(1234): Joined the cluster in 9msec 2018-10-08 03:44:32,447 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-08 03:49:10,361 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache(944): totalSize=747.70 KB, freeSize=994.87 MB, max=995.60 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2018-10-08 03:49:11,000 DEBUG [master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(427): data stats (chunk size=2097152): current pool size=0, created chunk count=3, reused chunk count=0, reuseRatio=0.00% 2018-10-08 03:49:11,000 DEBUG [master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(427): index stats (chunk size=209715): current pool size=0, created chunk count=0, reused chunk count=0, reuseRatio=0 2018-10-08 03:49:13,978 INFO [RS-EventLoopGroup-3-13] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:59878, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-08 03:49:13,989 WARN [master/asf904:0.Chore.1] master.CatalogJanitor(118): CatalogJanitor is disabled! Enabled=false, maintenanceMode=false, am=org.apache.hadoop.hbase.master.assignment.AssignmentManager@38a1f876, metaLoaded=true, hasRIT=false 2018-10-08 03:49:15,271 INFO [regionserver/asf904:0.Chore.1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1785): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 52481 ms 2018-10-08 03:49:16,308 INFO [regionserver/asf904:0.Chore.1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1785): MemstoreFlusherChore requesting flush of hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. because info has an old edit so flush to free WALs after random delay 57337 ms 2018-10-08 03:49:19,126 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-10-08 03:50:07,755 DEBUG [MemStoreFlusher.0] regionserver.FlushAllLargeStoresPolicy(70): Since none of the CFs were above the size, flushing all. 2018-10-08 03:50:07,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2647): Flushing 3/3 column families, dataSize=20.90 KB heapSize=33.89 KB 2018-10-08 03:50:07,850 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741856_1032{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 13436 2018-10-08 03:50:08,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=20.43 KB at sequenceid=56 (bloomFilter=false), to=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/.tmp/info/e905a04f78b0495d925bf277f5ee9ac3 2018-10-08 03:50:08,306 INFO [MemStoreFlusher.0] regionserver.StoreFileReader(608): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e905a04f78b0495d925bf277f5ee9ac3 2018-10-08 03:50:08,376 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741857_1033{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 4942 2018-10-08 03:50:08,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=476 B at sequenceid=56 (bloomFilter=false), to=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/.tmp/table/92880d965fd943b1924a207d877d56e0 2018-10-08 03:50:08,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/.tmp/info/e905a04f78b0495d925bf277f5ee9ac3 as hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/info/e905a04f78b0495d925bf277f5ee9ac3 2018-10-08 03:50:08,810 INFO [MemStoreFlusher.0] regionserver.StoreFileReader(608): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e905a04f78b0495d925bf277f5ee9ac3 2018-10-08 03:50:08,811 INFO [MemStoreFlusher.0] regionserver.HStore(1071): Added hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/info/e905a04f78b0495d925bf277f5ee9ac3, entries=62, sequenceid=56, filesize=13.1 K 2018-10-08 03:50:08,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/.tmp/table/92880d965fd943b1924a207d877d56e0 as hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/table/92880d965fd943b1924a207d877d56e0 2018-10-08 03:50:08,836 INFO [MemStoreFlusher.0] regionserver.HStore(1071): Added hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/meta/1588230740/table/92880d965fd943b1924a207d877d56e0, entries=5, sequenceid=56, filesize=4.8 K 2018-10-08 03:50:08,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2856): Finished flush of dataSize ~20.90 KB/21400, heapSize ~33.61 KB/34416, currentSize=0 B/0 for 1588230740 in 1085ms, sequenceid=56, compaction requested=false 2018-10-08 03:50:08,843 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(124): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2018-10-08 03:50:08,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2362): Flush status journal: Acquiring readlock on region at 1538970607753 Running coprocessor pre-flush hooks at 1538970607754 Obtaining lock to block concurrent updates at 1538970607755 Preparing flush snapshotting stores in 1588230740 at 1538970607755 Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=21400, getHeapSize=34656, getOffHeapSize=0 at 1538970607768 Flushing stores of hbase:meta,,1.1588230740 at 1538970607770 Flushing info: creating writer at 1538970607772 Flushing info: appending metadata at 1538970607815 Flushing info: closing flushed file at 1538970607816 Flushing table: creating writer at 1538970608352 Flushing table: appending metadata at 1538970608360 Flushing table: closing flushed file at 1538970608360 Flushing info: reopening flushed file at 1538970608798 Flushing table: reopening flushed file at 1538970608823 Finished flush of dataSize ~20.90 KB/21400, heapSize ~33.61 KB/34416, currentSize=0 B/0 for 1588230740 in 1085ms, sequenceid=56, compaction requested=false at 1538970608840 Running post-flush coprocessor hooks at 1538970608846 Flush successful at 1538970608846 2018-10-08 03:50:13,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2647): Flushing 1/1 column families, dataSize=78 B heapSize=488 B 2018-10-08 03:50:13,670 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|FINALIZED]]} size 0 2018-10-08 03:50:13,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/.tmp/info/cd6086033f304b01bff8272000fa4b9f 2018-10-08 03:50:13,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/.tmp/info/cd6086033f304b01bff8272000fa4b9f as hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/info/cd6086033f304b01bff8272000fa4b9f 2018-10-08 03:50:13,690 INFO [MemStoreFlusher.0] regionserver.HStore(1071): Added hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/data/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/info/cd6086033f304b01bff8272000fa4b9f, entries=2, sequenceid=6, filesize=4.8 K 2018-10-08 03:50:13,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2856): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e298524ff1440c61e0c0af4e0880daf3 in 46ms, sequenceid=6, compaction requested=false 2018-10-08 03:50:13,693 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(124): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2018-10-08 03:50:13,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2362): Flush status journal: Acquiring readlock on region at 1538970613647 Running coprocessor pre-flush hooks at 1538970613647 Obtaining lock to block concurrent updates at 1538970613648 Preparing flush snapshotting stores in e298524ff1440c61e0c0af4e0880daf3 at 1538970613648 Finished memstore snapshotting hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0 at 1538970613648 Flushing stores of hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. at 1538970613650 Flushing info: creating writer at 1538970613650 Flushing info: appending metadata at 1538970613659 Flushing info: closing flushed file at 1538970613659 Flushing info: reopening flushed file at 1538970613683 Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e298524ff1440c61e0c0af4e0880daf3 in 46ms, sequenceid=6, compaction requested=false at 1538970613693 Running post-flush coprocessor hooks at 1538970613694 Flush successful at 1538970613694 2018-10-08 03:54:10,361 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache(944): totalSize=747.70 KB, freeSize=994.87 MB, max=995.60 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2018-10-08 03:54:10,999 DEBUG [master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(427): data stats (chunk size=2097152): current pool size=3, created chunk count=3, reused chunk count=0, reuseRatio=0.00% 2018-10-08 03:54:11,000 DEBUG [master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(427): index stats (chunk size=209715): current pool size=0, created chunk count=0, reused chunk count=0, reuseRatio=0 2018-10-08 03:54:13,999 DEBUG [master/asf904:0.Chore.1] zookeeper.ZKUtil(704): replicationLogCleaner-0x16651c665170004, quorum=localhost:63142, baseZNode=/hbase Unable to get data of znode /hbase/replication/rs because node does not exist (not necessarily an error) 2018-10-08 03:54:14,000 DEBUG [master/asf904:0.Chore.1] replication.ZKReplicationQueueStorage(519): Didn't find a RegionServer that replicates, won't prevent deletions. 2018-10-08 03:54:14,381 INFO [RS-EventLoopGroup-3-15] ipc.ServerRpcConnection(556): Connection from 67.195.81.148:35310, version=3.0.0-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2018-10-08 03:54:14,406 WARN [master/asf904:0.Chore.1] master.CatalogJanitor(118): CatalogJanitor is disabled! Enabled=false, maintenanceMode=false, am=org.apache.hadoop.hbase.master.assignment.AssignmentManager@38a1f876, metaLoaded=true, hasRIT=false 2018-10-08 03:54:19,126 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties ====> TEST TIMED OUT. PRINTING THREAD DUMP. <==== Timestamp: 2018-10-08 03:57:03,118 "org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@6e8f2d99" daemon prio=5 tid=49 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) at java.lang.Thread.run(Thread.java:748) "MemStoreFlusher.0" daemon prio=5 tid=297 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:336) at java.lang.Thread.run(Thread.java:748) "Monitor thread for TaskMonitor" daemon prio=5 tid=232 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:302) at java.lang.Thread.run(Thread.java:748) "RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=47093" daemon prio=5 tid=223 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=38583" daemon prio=5 tid=209 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics" daemon prio=5 tid=238 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=47093" daemon prio=5 tid=224 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=47093" daemon prio=5 tid=217 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "Time-limited test" daemon prio=5 tid=23 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Threads.sleepWithoutInterrupt(Threads.java:166) at org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.waitProcedure(ProcedureTestingUtility.java:247) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.testRollbackAndDoubleExecution(MasterProcedureTestingUtility.java:500) at org.apache.hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure.testRollbackAndDoubleExecution(TestMergeTableRegionsProcedure.java:272) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) "IPC Server Responder" daemon prio=5 tid=145 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) at org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) "RS-EventLoopGroup-3-7" daemon prio=10 tid=317 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "IPC Parameter Sending Thread #2" daemon prio=5 tid=621 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "DataXceiver for client DFSClient_NONMAPREDUCE_-921593623_23 at /127.0.0.1:45528 [Receiving block BP-1232054963-67.195.81.148-1538970244597:blk_1073741832_1008]" daemon prio=5 tid=326 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:808) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47093" daemon prio=5 tid=219 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS_OPEN_REGION-regionserver/asf904:0-2" daemon prio=5 tid=382 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@5d96691c" daemon prio=5 tid=61 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:431) at java.lang.Thread.run(Thread.java:748) "Socket Reader #1 for port 54583" daemon prio=5 tid=42 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) "ReadOnlyZKClient-localhost:63142@0x4431d292" daemon prio=5 tid=344 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:313) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$66/1078432978.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) "IPC Server Responder" daemon prio=5 tid=44 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:982) at org.apache.hadoop.ipc.Server$Responder.run(Server.java:965) "SessionTracker" daemon prio=5 tid=178 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@6bd473d4" daemon prio=5 tid=45 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) at java.lang.Thread.run(Thread.java:748) "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data1)" daemon prio=5 tid=161 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628) "RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=47093" daemon prio=5 tid=221 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS-EventLoopGroup-3-14" daemon prio=10 tid=648 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "SplitLogWorker-asf904:47093" daemon prio=5 tid=301 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:461) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:219) at java.lang.Thread.run(Thread.java:748) "Thread-238" daemon prio=5 tid=566 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:523) "snapshot-hfile-cleaner-cache-refresher" daemon prio=5 tid=285 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38583" daemon prio=5 tid=197 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "Timer-0" daemon prio=5 tid=36 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=47093" daemon prio=5 tid=226 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS-EventLoopGroup-1-1" daemon prio=10 tid=189 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.util.JvmPauseMonitor$Monitor@62b5d6f3" daemon prio=5 tid=141 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:182) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 0 on 44213" daemon prio=5 tid=147 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RS_CLOSE_REGION-regionserver/asf904:0-0" daemon prio=5 tid=420 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-3" daemon prio=10 tid=306 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "Time-limited test-EventThread" daemon prio=5 tid=196 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) "RS-EventLoopGroup-3-6" daemon prio=10 tid=313 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-12" daemon prio=10 tid=601 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "RS_CLOSE_REGION-regionserver/asf904:0-1" daemon prio=5 tid=421 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@221c51da" daemon prio=5 tid=62 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4612) at java.lang.Thread.run(Thread.java:748) "RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=38583" daemon prio=5 tid=208 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47093" daemon prio=5 tid=222 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=38583" daemon prio=5 tid=204 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS-EventLoopGroup-3-8" daemon prio=10 tid=318 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "IPC Server listener on 44213" daemon prio=5 tid=142 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) "1560194700@qtp-1680263102-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34341" daemon prio=5 tid=74 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) "IPC Server handler 3 on 44213" daemon prio=5 tid=150 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "master/asf904:0:becomeActiveMaster-HFileCleaner.large.0-1538970252993" daemon prio=5 tid=286 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:250) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:219) "RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=47093" daemon prio=5 tid=227 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "CacheReplicationMonitor(1909427634)" daemon prio=5 tid=65 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) "IPC Server handler 9 on 54583" daemon prio=5 tid=60 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "master/asf904:0" daemon prio=5 tid=567 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.waitOnAssignQueue(AssignmentManager.java:1667) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1687) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:103) at org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1629) "ReadOnlyZKClient-localhost:63142@0x0caab251" daemon prio=5 tid=271 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:313) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$66/1078432978.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-1" daemon prio=10 tid=213 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=38583" daemon prio=5 tid=206 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "MemStoreFlusher.1" daemon prio=5 tid=299 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:336) at java.lang.Thread.run(Thread.java:748) "LeaseRenewer:jenkins.hfs.0@localhost:54583" daemon prio=5 tid=309 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) "refreshUsed-/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data1/current/BP-1232054963-67.195.81.148-1538970244597" daemon prio=5 tid=169 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-9" daemon prio=10 tid=325 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "ProcessThread(sid:0 cport:63142):" daemon prio=5 tid=180 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) "threadDeathWatcher-4-1" daemon prio=1 tid=288 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hbase.thirdparty.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "nioEventLoopGroup-2-1" prio=10 tid=76 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 5 on 54583" daemon prio=5 tid=56 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter@2c4c0fc7" daemon prio=5 tid=163 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl$LazyWriter.run(FsDatasetImpl.java:3088) at java.lang.Thread.run(Thread.java:748) "region-location-0" daemon prio=5 tid=342 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 5 on 44213" daemon prio=5 tid=152 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-7" daemon prio=5 tid=591 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "PacketResponder: BP-1232054963-67.195.81.148-1538970244597:blk_1073741830_1006, type=LAST_IN_PIPELINE, downstreams=0:[]" daemon prio=5 tid=312 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) at java.lang.Thread.run(Thread.java:748) "Socket Reader #1 for port 44213" daemon prio=5 tid=143 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:745) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:724) "Time-limited test.LruBlockCache.EvictionThread" daemon prio=5 tid=190 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:888) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber@69e2eed6" daemon prio=5 tid=64 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:4739) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 4 on 44213" daemon prio=5 tid=151 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38583" daemon prio=5 tid=199 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "Default-IPC-NioEventLoopGroup-5-2" daemon prio=10 tid=350 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "Time-limited test-SendThread(localhost:63142)" daemon prio=5 tid=195 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@41ebcc66" daemon prio=5 tid=63 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4656) at java.lang.Thread.run(Thread.java:748) "Time-limited test-EventThread" daemon prio=5 tid=215 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-9" daemon prio=5 tid=593 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server idle connection scanner for port 54583" daemon prio=5 tid=43 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "master/asf904:0:becomeActiveMaster-EventThread" daemon prio=5 tid=281 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) "surefire-forkedjvm-command-thread" daemon prio=5 tid=18 runnable java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115) at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391) at java.lang.Thread.run(Thread.java:748) "DecommissionMonitor-0" daemon prio=5 tid=50 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 2 on 44213" daemon prio=5 tid=149 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "Timer-1" daemon prio=5 tid=75 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "pool-4-thread-1" prio=5 tid=72 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-2" daemon prio=10 tid=278 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-5" daemon prio=5 tid=581 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "region-location-1" daemon prio=5 tid=343 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=47093" daemon prio=5 tid=229 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-1" daemon prio=5 tid=577 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=38583" daemon prio=5 tid=205 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "java.util.concurrent.ThreadPoolExecutor$Worker@3a9f1465[State = -1, empty queue]" daemon prio=5 tid=175 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Time-limited test-SendThread(localhost:63142)" daemon prio=5 tid=214 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) "HBase-Metrics2-1" daemon prio=5 tid=188 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-4" daemon prio=5 tid=580 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "process reaper" daemon prio=10 tid=69 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "regionserver/asf904:0.Chore.1" daemon prio=5 tid=293 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 1 on 54583" daemon prio=5 tid=52 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@1a6fecce" daemon prio=5 tid=71 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) at org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:146) at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:135) at java.lang.Thread.run(Thread.java:748) "RS_CLOSE_REGION-regionserver/asf904:0-2" daemon prio=5 tid=435 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS:0;asf904:47093-longCompactions-1538970253241" daemon prio=5 tid=292 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=38583" daemon prio=5 tid=210 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "WALProcedureStoreSyncThread" daemon prio=5 tid=563 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:809) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:106) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:308) "ReadOnlyZKClient-localhost:63142@0x73f6806c" daemon prio=5 tid=260 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:313) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$66/1078432978.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) "RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=38583" daemon prio=5 tid=207 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "Thread-98" daemon prio=5 tid=259 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:523) "OldWALsCleaner-1" daemon prio=5 tid=283 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:181) at org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$0(LogCleaner.java:159) at org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$119/1576100743.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=47093" daemon prio=5 tid=220 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS-EventLoopGroup-1-2" daemon prio=10 tid=279 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 8 on 54583" daemon prio=5 tid=59 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "IPC Server handler 9 on 44213" daemon prio=5 tid=156 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "master/asf904:0:becomeActiveMaster-SendThread(localhost:63142)" daemon prio=5 tid=280 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) "RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=47093" daemon prio=5 tid=228 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=47093" daemon prio=5 tid=225 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "133099739@qtp-2017769425-0" daemon prio=5 tid=34 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) "RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=38583" daemon prio=5 tid=202 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-3" daemon prio=5 tid=579 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=47093" daemon prio=5 tid=218 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "SyncThread:0" daemon prio=5 tid=179 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) "MobFileCache #0" daemon prio=5 tid=194 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 6 on 44213" daemon prio=5 tid=153 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "ProcedureDispatcherTimeoutThread" daemon prio=5 tid=269 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:78) at org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher$TimeoutExecutorThread.run(RemoteProcedureDispatcher.java:294) "org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1b130f2f" daemon prio=5 tid=37 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3635) at java.lang.Thread.run(Thread.java:748) "ReplicationExecutor-0" daemon prio=5 tid=302 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-13" daemon prio=10 tid=602 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@6792c19d" daemon prio=5 tid=39 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:401) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-8" daemon prio=5 tid=592 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "M:0;asf904:38583" daemon prio=5 tid=211 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:67) at org.apache.hadoop.hbase.master.HMaster.waitForMasterActive(HMaster.java:688) at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:885) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:833) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:932) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:595) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=38583" daemon prio=5 tid=201 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "asf904:47093Replication Statistics #0" daemon prio=5 tid=303 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "2100222532@qtp-1680263102-0" daemon prio=5 tid=73 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) "Timer for 'HBase' metrics system" daemon prio=5 tid=651 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "RS_OPEN_REGION-regionserver/asf904:0-0" daemon prio=5 tid=378 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-6" daemon prio=5 tid=590 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-0" daemon prio=5 tid=576 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.hdfs.PeerCache@71723f99" daemon prio=5 tid=240 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-10" daemon prio=10 tid=328 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "AsyncFSWAL-0" daemon prio=5 tid=314 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=38583" daemon prio=5 tid=200 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "IPC Server handler 3 on 54583" daemon prio=5 tid=54 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RS_CLOSE_META-regionserver/asf904:0-0" daemon prio=5 tid=308 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Client (1494280494) connection to localhost/127.0.0.1:54583 from jenkins" daemon prio=5 tid=157 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) "RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=47093" daemon prio=5 tid=216 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "AsyncFSWAL-0" daemon prio=5 tid=329 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Block report processor" daemon prio=5 tid=38 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.processQueue(BlockManager.java:3860) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.run(BlockManager.java:3849) "JvmPauseMonitor" daemon prio=5 tid=291 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:154) at java.lang.Thread.run(Thread.java:748) "RS_COMPACTED_FILES_DISCHARGER-regionserver/asf904:0-2" daemon prio=5 tid=578 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "ProcExecTimeout" daemon prio=5 tid=564 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.DelayQueue.take(DelayQueue.java:223) at org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:78) at org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:56) "regionserver/asf904:0.procedureResultReporter" daemon prio=5 tid=296 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:75) "RpcClient-timer-pool1-t1" daemon prio=5 tid=263 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:560) at org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:459) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 7 on 44213" daemon prio=5 tid=154 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "IPC Server handler 1 on 44213" daemon prio=5 tid=148 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RS:0;asf904:47093" daemon prio=5 tid=230 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:67) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1016) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:184) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:130) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:341) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:165) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-1-3" daemon prio=10 tid=351 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "1731402903@qtp-2017769425-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43017" daemon prio=5 tid=35 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) "RS-EventLoopGroup-3-4" daemon prio=10 tid=307 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "LeaseRenewer:jenkins@localhost:54583" daemon prio=5 tid=183 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) at java.lang.Thread.run(Thread.java:748) "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:63142" daemon prio=5 tid=177 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 8 on 44213" daemon prio=5 tid=155 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "IPC Server handler 0 on 54583" daemon prio=5 tid=51 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RS-EventLoopGroup-3-5" daemon prio=10 tid=310 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "RS_OPEN_REGION-regionserver/asf904:0-1" daemon prio=5 tid=380 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-11" daemon prio=10 tid=349 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "Finalizer" daemon prio=8 tid=3 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) "master/asf904:0.splitLogManager..Chore.1" daemon prio=5 tid=257 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 7 on 54583" daemon prio=5 tid=58 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "Signal Dispatcher" daemon prio=9 tid=4 runnable java.lang.Thread.State: RUNNABLE "surefire-forkedjvm-ping-30s" daemon prio=5 tid=19 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "VolumeScannerThread(/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data2)" daemon prio=5 tid=162 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628) "PEWorker-1" daemon prio=5 tid=565 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:159) at org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:141) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1998) "IPC Server handler 4 on 54583" daemon prio=5 tid=55 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "IPC Server idle connection scanner for port 44213" daemon prio=5 tid=144 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "LruBlockCacheStatsExecutor" daemon prio=5 tid=192 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "IPC Server handler 2 on 54583" daemon prio=5 tid=53 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "Idle-Rpc-Conn-Sweeper-pool2-t1" daemon prio=5 tid=264 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Default-IPC-NioEventLoopGroup-5-1" daemon prio=10 tid=348 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "regionserver/asf904:0.leaseChecker" daemon prio=5 tid=294 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:95) at java.lang.Thread.run(Thread.java:748) "refreshUsed-/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data2/current/BP-1232054963-67.195.81.148-1538970244597" daemon prio=5 tid=168 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:132) at java.lang.Thread.run(Thread.java:748) "master/asf904:0:becomeActiveMaster-HFileCleaner.small.0-1538970252994" daemon prio=5 tid=287 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:550) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:250) at org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:234) "RS_OPEN_PRIORITY_REGION-regionserver/asf904:0-0" daemon prio=5 tid=337 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "master/asf904:0:becomeActiveMaster-MemStoreChunkPool Statistics" daemon prio=5 tid=236 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "RegionServerTracker-0" daemon prio=5 tid=289 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "Reference Handler" daemon prio=10 tid=2 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "IPC Server handler 6 on 54583" daemon prio=5 tid=57 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:136) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2185) "RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=38583" daemon prio=5 tid=203 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests_master-M6IJIUBEFGBLJA2QZFJ5NFMN4UCZCOJZHSKUZTUZSLWEH6TJXWBQ/hbase-server/target/test-data/217fe657-ba5f-1903-6385-0955101ad04d/cluster_8c222a20-77cc-33cb-d406-64340eeba8d0/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:54583" daemon prio=5 tid=146 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:130) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:542) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:659) at java.lang.Thread.run(Thread.java:748) "pool-2-thread-1" prio=5 tid=33 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=24 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3060) at java.lang.Thread.run(Thread.java:748) "PacketResponder: BP-1232054963-67.195.81.148-1538970244597:blk_1073741832_1008, type=LAST_IN_PIPELINE, downstreams=0:[]" daemon prio=5 tid=327 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) at java.lang.Thread.run(Thread.java:748) "RS-EventLoopGroup-3-15" daemon prio=10 tid=649 runnable java.lang.Thread.State: RUNNABLE at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "DataXceiver for client DFSClient_NONMAPREDUCE_-921593623_23 at /127.0.0.1:45512 [Receiving block BP-1232054963-67.195.81.148-1538970244597:blk_1073741830_1006]" daemon prio=5 tid=311 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:808) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) at java.lang.Thread.run(Thread.java:748) "master/asf904:0.Chore.1" daemon prio=5 tid=284 timed_waiting java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) "OldWALsCleaner-0" daemon prio=5 tid=282 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:181) at org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$0(LogCleaner.java:159) at org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$119/1576100743.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) "RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38583" daemon prio=5 tid=198 in Object.wait() java.lang.Thread.State: WAITING (on object monitor) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) at org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "IPC Server listener on 54583" daemon prio=5 tid=41 runnable java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.apache.hadoop.ipc.Server$Listener.run(Server.java:807) "regionserver/asf904:0.logRoller" daemon prio=5 tid=295 timed_waiting java.lang.Thread.State: TIMED_WAITING at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:167) at java.lang.Thread.run(Thread.java:748) "main" prio=5 tid=1 runnable java.lang.Thread.State: RUNNABLE at java.lang.Thread.dumpThreads(Native Method) at java.lang.Thread.getAllStackTraces(Thread.java:1610) at org.apache.hadoop.hbase.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:88) at org.apache.hadoop.hbase.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:74) at org.apache.hadoop.hbase.TimedOutTestsListener.testFailure(TimedOutTestsListener.java:62) at org.junit.runner.notification.SynchronizedRunListener.testFailure(SynchronizedRunListener.java:63) at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:142) at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) at org.junit.runner.notification.RunNotifier.fireTestFailures(RunNotifier.java:138) at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:132) at org.apache.maven.surefire.common.junit4.Notifier.fireTestFailure(Notifier.java:114) at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:23) at org.junit.internal.runners.model.EachTestNotifier.addMultipleFailureException(EachTestNotifier.java:29) at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:21) at org.junit.runners.ParentRunner.run(ParentRunner.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) 2018-10-08 03:57:03,166 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5842ade2 2018-10-08 03:57:03,167 INFO [Thread-3] regionserver.HRegionServer(2160): ***** STOPPING region server 'asf904.gq1.ygridcore.net,47093,1538970250889' ***** 2018-10-08 03:57:03,167 INFO [Thread-3] regionserver.HRegionServer(2174): STOPPED: Shutdown hook 2018-10-08 03:57:03,167 INFO [RS:0;asf904:47093] regionserver.SplitLogWorker(241): Sending interrupt to stop the worker thread 2018-10-08 03:57:03,167 INFO [RS:0;asf904:47093] regionserver.HeapMemoryManager(221): Stopping 2018-10-08 03:57:03,168 INFO [SplitLogWorker-asf904:47093] regionserver.SplitLogWorker(223): SplitLogWorker interrupted. Exiting. 2018-10-08 03:57:03,168 INFO [RS:0;asf904:47093] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-10-08 03:57:03,168 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-10-08 03:57:03,168 INFO [SplitLogWorker-asf904:47093] regionserver.SplitLogWorker(232): SplitLogWorker asf904.gq1.ygridcore.net,47093,1538970250889 exiting 2018-10-08 03:57:03,168 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-10-08 03:57:03,168 INFO [RS:0;asf904:47093] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-10-08 03:57:03,170 INFO [RS:0;asf904:47093] regionserver.HRegionServer(1070): stopping server asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:57:03,171 DEBUG [RS:0;asf904:47093] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-10-08 03:57:03,171 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1554): Closing e298524ff1440c61e0c0af4e0880daf3, disabling compactions & flushes 2018-10-08 03:57:03,172 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1554): Closing 8f69cb83fe1220eb0f370056e2b218e8, disabling compactions & flushes 2018-10-08 03:57:03,172 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1594): Updates disabled for region hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:57:03,172 INFO [RS:0;asf904:47093] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x0caab251 to localhost:63142 2018-10-08 03:57:03,172 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 381466ba5ddfd0f7f6d889dbd8abea87, disabling compactions & flushes 2018-10-08 03:57:03,173 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:57:03,173 DEBUG [RS:0;asf904:47093] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-08 03:57:03,172 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1594): Updates disabled for region testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:57:03,174 INFO [RS:0;asf904:47093] regionserver.CompactSplit(431): Waiting for Split Thread to finish... 2018-10-08 03:57:03,174 INFO [RS:0;asf904:47093] regionserver.CompactSplit(431): Waiting for Large Compaction Thread to finish... 2018-10-08 03:57:03,174 INFO [RS:0;asf904:47093] regionserver.CompactSplit(431): Waiting for Small Compaction Thread to finish... 2018-10-08 03:57:03,175 INFO [RS:0;asf904:47093] regionserver.HRegionServer(1400): Waiting on 4 regions to close 2018-10-08 03:57:03,175 DEBUG [RS:0;asf904:47093] regionserver.HRegionServer(1404): Online Regions={8f69cb83fe1220eb0f370056e2b218e8=testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8., 1588230740=hbase:meta,,1.1588230740, 381466ba5ddfd0f7f6d889dbd8abea87=testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87., e298524ff1440c61e0c0af4e0880daf3=hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3.} 2018-10-08 03:57:03,176 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1554): Closing 1588230740, disabling compactions & flushes 2018-10-08 03:57:03,176 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1594): Updates disabled for region hbase:meta,,1.1588230740 2018-10-08 03:57:03,197 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/meta/1588230740/recovered.edits/59.seqid, newMaxSeqId=59, maxSeqId=1 2018-10-08 03:57:03,197 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/hbase/namespace/e298524ff1440c61e0c0af4e0880daf3/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2018-10-08 03:57:03,197 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/8f69cb83fe1220eb0f370056e2b218e8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:57:03,198 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] wal.WALSplitter(696): Wrote file=hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/default/testRollbackAndDoubleExecution/381466ba5ddfd0f7f6d889dbd8abea87/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2018-10-08 03:57:03,198 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-10-08 03:57:03,203 INFO [RS_CLOSE_REGION-regionserver/asf904:0-2] regionserver.HRegion(1711): Closed hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:57:03,204 INFO [RS_CLOSE_REGION-regionserver/asf904:0-1] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:57:03,206 INFO [RS_CLOSE_META-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed hbase:meta,,1.1588230740 2018-10-08 03:57:03,204 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-2] handler.CloseRegionHandler(129): Closed hbase:namespace,,1538970254013.e298524ff1440c61e0c0af4e0880daf3. 2018-10-08 03:57:03,207 DEBUG [RS_CLOSE_META-regionserver/asf904:0-0] handler.CloseRegionHandler(129): Closed hbase:meta,,1.1588230740 2018-10-08 03:57:03,207 INFO [RS_CLOSE_REGION-regionserver/asf904:0-0] regionserver.HRegion(1711): Closed testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:57:03,206 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-1] handler.CloseRegionHandler(129): Closed testRollbackAndDoubleExecution,2,1538970262521.8f69cb83fe1220eb0f370056e2b218e8. 2018-10-08 03:57:03,208 DEBUG [RS_CLOSE_REGION-regionserver/asf904:0-0] handler.CloseRegionHandler(129): Closed testRollbackAndDoubleExecution,1,1538970262521.381466ba5ddfd0f7f6d889dbd8abea87. 2018-10-08 03:57:03,321 INFO [regionserver/asf904:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-10-08 03:57:03,321 INFO [regionserver/asf904:0.Chore.1] hbase.ScheduledChore(180): Chore: CompactionChecker was stopped 2018-10-08 03:57:03,376 INFO [RS:0;asf904:47093] regionserver.HRegionServer(1098): stopping server asf904.gq1.ygridcore.net,47093,1538970250889; all regions closed. 2018-10-08 03:57:03,384 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 0 2018-10-08 03:57:03,394 INFO [regionserver/asf904:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-10-08 03:57:03,395 DEBUG [RS:0;asf904:47093] wal.AbstractFSWAL(858): Moved 1 WAL file(s) to /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/oldWALs 2018-10-08 03:57:03,395 INFO [RS:0;asf904:47093] wal.AbstractFSWAL(861): Closed WAL: AsyncFSWAL asf904.gq1.ygridcore.net%2C47093%2C1538970250889.meta:.meta(num 1538970253488) 2018-10-08 03:57:03,400 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 7241 2018-10-08 03:57:03,400 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889/asf904.gq1.ygridcore.net%2C47093%2C1538970250889.1538970254360 not finished, retry = 0 2018-10-08 03:57:03,505 DEBUG [RS:0;asf904:47093] wal.AbstractFSWAL(858): Moved 1 WAL file(s) to /user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/oldWALs 2018-10-08 03:57:03,505 INFO [RS:0;asf904:47093] wal.AbstractFSWAL(861): Closed WAL: AsyncFSWAL asf904.gq1.ygridcore.net%2C47093%2C1538970250889:(num 1538970254360) 2018-10-08 03:57:03,505 DEBUG [RS:0;asf904:47093] ipc.AbstractRpcClient(483): Stopping rpc client 2018-10-08 03:57:03,505 INFO [RS:0;asf904:47093] regionserver.Leases(149): Closed leases 2018-10-08 03:57:03,508 INFO [RS:0;asf904:47093] hbase.ChoreService(327): Chore service for: regionserver/asf904:0 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-10-08 03:57:03,508 INFO [regionserver/asf904:0.logRoller] regionserver.LogRoller(222): LogRoller exiting. 2018-10-08 03:57:03,516 INFO [RS:0;asf904:47093] ipc.NettyRpcServer(144): Stopping server on /67.195.81.148:47093 2018-10-08 03:57:03,526 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:38583-0x16651c665170000, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-08 03:57:03,526 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf904.gq1.ygridcore.net,47093,1538970250889 2018-10-08 03:57:03,526 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:47093-0x16651c665170001, quorum=localhost:63142, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-10-08 03:57:03,527 INFO [RegionServerTracker-0] master.RegionServerTracker(168): RegionServer ephemeral node deleted, processing expiration [asf904.gq1.ygridcore.net,47093,1538970250889] 2018-10-08 03:57:03,527 INFO [RegionServerTracker-0] master.ServerManager(604): Processing expiration of asf904.gq1.ygridcore.net,47093,1538970250889 on asf904.gq1.ygridcore.net,38583,1538970249417 2018-10-08 03:57:03,529 INFO [RS:0;asf904:47093] regionserver.HRegionServer(1154): Exiting; stopping=asf904.gq1.ygridcore.net,47093,1538970250889; zookeeper connection closed. 2018-10-08 03:57:03,530 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@12547a9c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(222): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@12547a9c 2018-10-08 03:57:03,532 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-10-08 03:57:03,543 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 0 2018-10-08 03:57:03,545 WARN [WALProcedureStoreSyncThread] wal.WALProcedureStore(887): unable to sync slots, retry=0 2018-10-08 03:57:03,545 WARN [WALProcedureStoreSyncThread] wal.WALProcedureStore(887): unable to sync slots, retry=1 2018-10-08 03:57:03,545 WARN [WALProcedureStoreSyncThread] wal.WALProcedureStore(887): unable to sync slots, retry=2 2018-10-08 03:57:03,549 WARN [WALProcedureStoreSyncThread] wal.WALProcedureStore(1123): Unable to write the trailer java.nio.channels.ClosedChannelException at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1732) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:2985) at org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.flushIfNotAvailable(CodedOutputStream.java:2980) at org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream$OutputStreamEncoder.writeInt32(CodedOutputStream.java:2659) at org.apache.hbase.thirdparty.com.google.protobuf.CodedOutputStream.writeEnum(CodedOutputStream.java:301) at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$ProcedureWALEntry.writeTo(ProcedureProtos.java:7700) at org.apache.hbase.thirdparty.com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:98) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeTrailer(ProcedureWALFormat.java:130) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.closeCurrentLogStream(WALProcedureStore.java:1120) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1088) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1026) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriterWithRetries(WALProcedureStore.java:938) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncSlots(WALProcedureStore.java:894) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:845) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:106) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:308) 2018-10-08 03:57:03,549 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore(1107): Rolled new Procedure Store WAL, id=14 2018-10-08 03:57:03,554 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741860_1036{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-504f7cce-a4b4-4bf8-b776-e111c3045e3a:NORMAL:127.0.0.1:41173|RBW]]} size 211 2018-10-08 03:57:03,566 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1124): Stored pid=47, state=RUNNABLE:SERVER_CRASH_START, hasLock=false; ServerCrashProcedure server=asf904.gq1.ygridcore.net,47093,1538970250889, splitWal=true, meta=true 2018-10-08 03:57:03,567 DEBUG [RegionServerTracker-0] assignment.AssignmentManager(1330): Added=asf904.gq1.ygridcore.net,47093,1538970250889 to dead servers, submitted shutdown handler to be executed meta=true 2018-10-08 03:57:03,570 INFO [PEWorker-1] procedure.ServerCrashProcedure(131): Start pid=47, state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure server=asf904.gq1.ygridcore.net,47093,1538970250889, splitWal=true, meta=true 2018-10-08 03:57:03,570 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(119): Toggle KILL before store update to: true 2018-10-08 03:57:03,573 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(207): Splitting meta WALs pid=47, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, hasLock=true; ServerCrashProcedure server=asf904.gq1.ygridcore.net,47093,1538970250889, splitWal=true, meta=true 2018-10-08 03:57:03,576 DEBUG [PEWorker-1] master.MasterWalManager(312): Renamed region directory: hdfs://localhost:54583/user/jenkins/test-data/4f8a17b0-915f-acad-51a8-b1569c687904/WALs/asf904.gq1.ygridcore.net,47093,1538970250889-splitting 2018-10-08 03:57:03,577 INFO [PEWorker-1] master.SplitLogManager(465): dead splitlog workers [asf904.gq1.ygridcore.net,47093,1538970250889] 2018-10-08 03:57:03,960 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41173 is added to blk_1073741861_1037{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-729ccf3f-b466-40ab-a7ad-c0582ea36cf3:NORMAL:127.0.0.1:41173|RBW]]} size 1337 2018-10-08 03:57:04,362 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.