2018-01-05 21:49:08,870 INFO [main] hbase.HBaseTestingUtility(945): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2018-01-05 21:49:08,939 INFO [main] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/cluster_9b253a66-d5a8-41fd-bb42-b9969286e348, deleteOnExit=true 2018-01-05 21:49:08,939 INFO [main] hbase.HBaseTestingUtility(960): STARTING DFS 2018-01-05 21:49:08,940 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/cache_data in system properties and HBase conf 2018-01-05 21:49:08,940 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/hadoop_tmp in system properties and HBase conf 2018-01-05 21:49:08,941 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/hadoop_logs in system properties and HBase conf 2018-01-05 21:49:08,942 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/mapred_local in system properties and HBase conf 2018-01-05 21:49:08,943 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/mapred_temp in system properties and HBase conf 2018-01-05 21:49:08,944 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2018-01-05 21:49:09,253 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-01-05 21:49:09,972 DEBUG [main] fs.HFileSystem(318): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-01-05 21:49:11,302 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-01-05 21:49:11,572 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-01-05 21:49:11,725 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-05 21:49:11,777 INFO [main] log.Slf4jLog(67): Extract jar:file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/%7BWORKSPACE%7D/maven-repo/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_52769_hdfs____.qecteh/webapp 2018-01-05 21:49:12,087 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52769 2018-01-05 21:49:13,479 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-05 21:49:13,581 INFO [main] log.Slf4jLog(67): Extract jar:file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/%7BWORKSPACE%7D/maven-repo/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_44482_datanode____2i2umt/webapp 2018-01-05 21:49:13,864 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44482 2018-01-05 21:49:16,156 ERROR [DataNode: [[[DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/cluster_9b253a66-d5a8-41fd-bb42-b9969286e348/dfs/data/data1/, [DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/1f274f10-87e2-4f25-b7c3-fc081a1a5b16/cluster_9b253a66-d5a8-41fd-bb42-b9969286e348/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:42367] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-05 21:49:16,361 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1f85138e23384d: from storage DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71 node DatanodeRegistration(127.0.0.1:51755, datanodeUuid=e54472d2-51e6-434f-ab21-a0f4c8b714f3, infoPort=58655, infoSecurePort=0, ipcPort=47231, storageInfo=lv=-56;cid=testClusterID;nsid=1605144140;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs 2018-01-05 21:49:16,365 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1f85138e23384d: from storage DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3 node DatanodeRegistration(127.0.0.1:51755, datanodeUuid=e54472d2-51e6-434f-ab21-a0f4c8b714f3, infoPort=58655, infoSecurePort=0, ipcPort=47231, storageInfo=lv=-56;cid=testClusterID;nsid=1605144140;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-05 21:49:16,587 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-05 21:49:16,630 INFO [main] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=50569 2018-01-05 21:49:16,682 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:16,691 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:17,343 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 7 2018-01-05 21:49:17,758 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc with version=8 2018-01-05 21:49:17,759 INFO [main] hbase.HBaseTestingUtility(1204): Setting hbase.fs.tmp.dir to hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/hbase-staging 2018-01-05 21:49:18,158 INFO [main] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-01-05 21:49:18,535 INFO [main] client.ConnectionUtils(122): master/instance-2.c.gcp-hbase.internal/10.128.0.3:0 server-side Connection retries=45 2018-01-05 21:49:18,561 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-05 21:49:18,565 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-05 21:49:18,565 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-05 21:49:18,851 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-05 21:49:19,490 DEBUG [main] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-01-05 21:49:19,625 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/10.128.0.3:37786 2018-01-05 21:49:19,637 INFO [main] hfile.CacheConfig(553): Allocating On heap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-01-05 21:49:19,689 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:19,690 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:19,696 DEBUG [main] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-01-05 21:49:19,698 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:19,703 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:19,795 INFO [main] zookeeper.RecoverableZooKeeper(107): Process identifier=master:37786 connecting to ZooKeeper ensemble=localhost:50569 2018-01-05 21:49:19,928 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:377860x0, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 21:49:19,932 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:37786-0x160c84cad8a0000 connected 2018-01-05 21:49:20,009 DEBUG [main] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-05 21:49:20,011 DEBUG [main] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 21:49:20,041 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=37786 2018-01-05 21:49:20,061 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=37786 2018-01-05 21:49:20,090 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=37786 2018-01-05 21:49:20,114 INFO [main] master.HMaster(476): hbase.rootdir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc, hbase.cluster.distributed=false 2018-01-05 21:49:20,489 INFO [main] master.HMaster(1987): Adding backup master ZNode /hbase/backup-masters/instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:20,506 DEBUG [main] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:20,619 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-05 21:49:20,626 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 21:49:20,627 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-05 21:49:20,627 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 21:49:20,628 INFO [instance-2:37786.masterManager] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/instance-2.c.gcp-hbase.internal,37786,1515188957846 from backup master directory 2018-01-05 21:49:20,634 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:20,635 WARN [instance-2:37786.masterManager] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-05 21:49:20,635 INFO [instance-2:37786.masterManager] master.ActiveMasterManager(181): Registered Active Master=instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:20,653 INFO [instance-2:37786.masterManager] regionserver.ChunkCreator(408): Allocating MemStoreChunkPool with chunk size 2 MB, max count 497, initial count 0 2018-01-05 21:49:20,715 INFO [main] client.ConnectionUtils(122): regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0 server-side Connection retries=45 2018-01-05 21:49:20,723 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-05 21:49:20,723 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-05 21:49:20,724 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-05 21:49:20,733 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-05 21:49:20,736 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-05 21:49:20,766 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/10.128.0.3:45565 2018-01-05 21:49:20,767 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:20,768 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:20,770 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:20,794 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:20,803 INFO [main] zookeeper.RecoverableZooKeeper(107): Process identifier=regionserver:45565 connecting to ZooKeeper ensemble=localhost:50569 2018-01-05 21:49:20,832 DEBUG [main] zookeeper.ZKUtil(355): regionserver:455650x0, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 21:49:20,833 DEBUG [main] zookeeper.ZKUtil(357): regionserver:455650x0, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 21:49:20,834 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=45565 2018-01-05 21:49:20,836 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:455650x0, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 21:49:20,847 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:45565-0x160c84cad8a0001 connected 2018-01-05 21:49:20,868 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=45565 2018-01-05 21:49:20,913 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=45565 2018-01-05 21:49:21,024 INFO [M:0;instance-2:37786] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x750df039 to localhost:50569, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 21:49:21,033 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:21,029 INFO [RS:0;instance-2:45565] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x50e6d448 to localhost:50569, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 21:49:21,070 DEBUG [instance-2:37786.masterManager] util.FSUtils(665): Created cluster ID file at hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/hbase.id with ID: e44adf2e-f66b-4f88-aaca-36a4af54fb8b 2018-01-05 21:49:21,140 WARN [M:0;instance-2:37786] client.ConnectionImplementation(528): Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:132) at org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) at org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:775) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:806) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:821) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:932) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:546) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) ... 1 more 2018-01-05 21:49:21,157 DEBUG [M:0;instance-2:37786] client.ConnectionImplementation(532): clusterid came back null, using default default-cluster 2018-01-05 21:49:21,142 WARN [RS:0;instance-2:45565] client.ConnectionImplementation(528): Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:132) at org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) at org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:775) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:806) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:821) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:932) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) ... 1 more 2018-01-05 21:49:21,162 DEBUG [RS:0;instance-2:45565] client.ConnectionImplementation(532): clusterid came back null, using default default-cluster 2018-01-05 21:49:21,206 DEBUG [RS:0;instance-2:45565] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e260c7f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 21:49:21,207 DEBUG [M:0;instance-2:37786] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4609a18c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 21:49:21,226 INFO [instance-2:37786.masterManager] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-05 21:49:21,243 INFO [instance-2:37786.masterManager] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc Table name == hbase:meta 2018-01-05 21:49:21,319 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:21,335 DEBUG [instance-2:37786.masterManager] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-05 21:49:21,435 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/info 2018-01-05 21:49:21,479 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:21,486 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:21,507 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:21,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_barrier 2018-01-05 21:49:21,557 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:21,558 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:21,559 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:21,566 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_meta 2018-01-05 21:49:21,567 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:21,567 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:21,568 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:21,589 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_position 2018-01-05 21:49:21,590 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:21,590 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:21,609 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:21,634 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/table 2018-01-05 21:49:21,635 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:21,635 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:21,636 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:21,657 DEBUG [instance-2:37786.masterManager] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740 2018-01-05 21:49:21,696 DEBUG [instance-2:37786.masterManager] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-05 21:49:21,707 DEBUG [instance-2:37786.masterManager] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 21:49:21,707 INFO [instance-2:37786.masterManager] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-05 21:49:21,707 DEBUG [instance-2:37786.masterManager] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-05 21:49:21,708 DEBUG [instance-2:37786.masterManager] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-05 21:49:21,714 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-05 21:49:21,716 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-05 21:49:21,718 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-05 21:49:21,718 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-05 21:49:21,718 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-05 21:49:21,719 INFO [instance-2:37786.masterManager] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-05 21:49:21,861 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:21,875 DEBUG [instance-2:37786.masterManager] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-05 21:49:21,959 INFO [instance-2:37786.masterManager] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 21:49:21,995 INFO [instance-2:37786.masterManager] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-05 21:49:21,998 DEBUG [instance-2:37786.masterManager] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-05 21:49:22,126 INFO [instance-2:37786.masterManager] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-05 21:49:22,137 INFO [instance-2:37786.masterManager] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-01-05 21:49:22,143 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-05 21:49:22,147 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-05 21:49:22,171 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-05 21:49:22,175 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-05 21:49:22,219 INFO [instance-2:37786.masterManager] assignment.AssignmentManager(219): Starting assignment manager 2018-01-05 21:49:22,315 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-05 21:49:22,316 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-05 21:49:22,321 INFO [RS:0;instance-2:45565] regionserver.HRegionServer(867): ClusterId : e44adf2e-f66b-4f88-aaca-36a4af54fb8b 2018-01-05 21:49:22,325 INFO [M:0;instance-2:37786] regionserver.HRegionServer(867): ClusterId : e44adf2e-f66b-4f88-aaca-36a4af54fb8b 2018-01-05 21:49:22,327 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-05 21:49:22,330 INFO [instance-2:37786.masterManager] master.HMaster(728): Server active/primary master=instance-2.c.gcp-hbase.internal,37786,1515188957846, sessionid=0x160c84cad8a0000, setting cluster-up flag (Was=false) 2018-01-05 21:49:22,391 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-05 21:49:22,396 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-05 21:49:22,401 DEBUG [instance-2:37786.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/flush-table-proc/acquired already exists 2018-01-05 21:49:22,412 INFO [instance-2:37786.masterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-05 21:49:22,417 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-05 21:49:22,418 DEBUG [RS:0;instance-2:45565] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d6da230, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=instance-2.c.gcp-hbase.internal/10.128.0.3:0 2018-01-05 21:49:22,422 DEBUG [RS:0;instance-2:45565] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;instance-2:45565 2018-01-05 21:49:22,428 DEBUG [instance-2:37786.masterManager] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:22,444 DEBUG [instance-2:37786.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/online-snapshot/acquired already exists 2018-01-05 21:49:22,450 INFO [instance-2:37786.masterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-05 21:49:22,455 DEBUG [instance-2:37786.masterManager] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:instance-2.c.gcp-hbase.internal,37786,1515188957846 2018-01-05 21:49:22,468 INFO [RS:0;instance-2:45565] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-05 21:49:22,478 INFO [RS:0;instance-2:45565] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-05 21:49:22,482 INFO [RS:0;instance-2:45565] regionserver.HRegionServer(2620): reportForDuty to master=instance-2.c.gcp-hbase.internal,37786,1515188957846 with port=45565, startcode=1515188960705 2018-01-05 21:49:22,572 INFO [instance-2:37786.masterManager] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-05 21:49:22,595 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-instance-2:37786, corePoolSize=5, maxPoolSize=5 2018-01-05 21:49:22,597 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-instance-2:37786, corePoolSize=5, maxPoolSize=5 2018-01-05 21:49:22,597 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-instance-2:37786, corePoolSize=5, maxPoolSize=5 2018-01-05 21:49:22,597 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-instance-2:37786, corePoolSize=5, maxPoolSize=5 2018-01-05 21:49:22,598 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-instance-2:37786, corePoolSize=10, maxPoolSize=10 2018-01-05 21:49:22,601 DEBUG [instance-2:37786.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-instance-2:37786, corePoolSize=1, maxPoolSize=1 2018-01-05 21:49:22,660 DEBUG [RS:0;instance-2:45565] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:37786 2018-01-05 21:49:22,676 INFO [instance-2:37786.masterManager] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-05 21:49:22,683 INFO [instance-2:37786.masterManager] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-05 21:49:22,714 DEBUG [instance-2:37786.masterManager] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-05 21:49:22,719 INFO [instance-2:37786.masterManager] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-05 21:49:22,721 INFO [instance-2:37786.masterManager] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 38msec 2018-01-05 21:49:22,722 DEBUG [instance-2:37786.masterManager] wal.WALProcedureStore(409): No state logs to replay. 2018-01-05 21:49:22,723 DEBUG [instance-2:37786.masterManager] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-05 21:49:22,723 INFO [instance-2:37786.masterManager] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 2msec 2018-01-05 21:49:22,739 INFO [instance-2:37786.masterManager] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515188992739 2018-01-05 21:49:22,749 INFO [instance-2:37786.masterManager] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-05 21:49:22,780 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-05 21:49:22,786 INFO [instance-2:37786.masterManager] zookeeper.RecoverableZooKeeper(107): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:50569 2018-01-05 21:49:22,804 DEBUG [instance-2:37786.masterManager-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 21:49:22,804 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-05 21:49:22,806 DEBUG [instance-2:37786.masterManager-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160c84cad8a0004 connected 2018-01-05 21:49:22,807 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-05 21:49:22,807 INFO [instance-2:37786.masterManager] cleaner.CleanerChore(113): Cleaner pool size is 1 2018-01-05 21:49:22,807 INFO [instance-2:37786.masterManager] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-05 21:49:22,828 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-05 21:49:22,845 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-05 21:49:22,846 DEBUG [instance-2:37786.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-05 21:49:22,861 DEBUG [instance-2:37786.masterManager] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[instance-2:37786.masterManager-HFileCleaner.large.0-1515188962848,5,main] 2018-01-05 21:49:22,864 DEBUG [instance-2:37786.masterManager] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[instance-2:37786.masterManager-HFileCleaner.small.0-1515188962861,5,main] 2018-01-05 21:49:22,913 INFO [instance-2:37786.masterManager] master.HMaster(824): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515188960611, completionTime=-1 2018-01-05 21:49:22,915 INFO [instance-2:37786.masterManager] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=1 server(s), max=1 server(s), timeout=4500ms, lastChange=0ms 2018-01-05 21:49:22,928 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.0 (auth:SIMPLE) 2018-01-05 21:49:22,938 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 39654 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:22,995 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.ServerManager(432): Registering server=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,016 INFO [instance-2:37786.masterManager] master.ServerManager(891): Finished wait on RegionServer count=1; waited=101ms, expected min=1 server(s), max=1 server(s), master is running 2018-01-05 21:49:23,029 DEBUG [RS:0;instance-2:45565] regionserver.HRegionServer(1495): Config from master: hbase.rootdir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc 2018-01-05 21:49:23,030 DEBUG [RS:0;instance-2:45565] regionserver.HRegionServer(1495): Config from master: fs.defaultFS=hdfs://localhost:42367 2018-01-05 21:49:23,030 DEBUG [RS:0;instance-2:45565] regionserver.HRegionServer(1495): Config from master: hbase.master.info.port=-1 2018-01-05 21:49:23,058 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-05 21:49:23,060 DEBUG [RS:0;instance-2:45565] zookeeper.ZKUtil(355): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,060 WARN [RS:0;instance-2:45565] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-05 21:49:23,061 DEBUG [RS:0;instance-2:45565] regionserver.HRegionServer(1801): logDir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,063 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,094 INFO [instance-2:37786.masterManager] master.HMaster(3513): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-05 21:49:23,117 INFO [RS:0;instance-2:45565] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-05 21:49:23,203 INFO [RS:0;instance-2:45565] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x1a900b62 to localhost:50569, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 21:49:23,276 DEBUG [RS:0;instance-2:45565] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d92147, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 21:49:23,277 DEBUG [RS:0;instance-2:45565] regionserver.Replication(152): ReplicationStatisticsThread 300 2018-01-05 21:49:23,299 INFO [RS:0;instance-2:45565] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-05 21:49:23,328 INFO [RS:0;instance-2:45565] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-05 21:49:23,334 INFO [RS:0;instance-2:45565] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-05 21:49:23,335 INFO [RS:0;instance-2:45565] regionserver.HRegionServer$CompactionChecker(1695): CompactionChecker runs every 1sec 2018-01-05 21:49:23,365 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-instance-2:45565, corePoolSize=3, maxPoolSize=3 2018-01-05 21:49:23,366 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-instance-2:45565, corePoolSize=1, maxPoolSize=1 2018-01-05 21:49:23,367 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-instance-2:45565, corePoolSize=3, maxPoolSize=3 2018-01-05 21:49:23,367 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-instance-2:45565, corePoolSize=3, maxPoolSize=3 2018-01-05 21:49:23,367 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-instance-2:45565, corePoolSize=1, maxPoolSize=1 2018-01-05 21:49:23,367 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-instance-2:45565, corePoolSize=2, maxPoolSize=2 2018-01-05 21:49:23,368 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-instance-2:45565, corePoolSize=10, maxPoolSize=10 2018-01-05 21:49:23,368 DEBUG [RS:0;instance-2:45565] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-instance-2:45565, corePoolSize=3, maxPoolSize=3 2018-01-05 21:49:23,453 DEBUG [instance-2:37786.masterManager] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 21:49:23,465 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 21:49:23,465 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 21:49:23,477 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-05 21:49:23,492 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-05 21:49:23,499 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 21:49:23,500 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-05 21:49:23,500 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-05 21:49:23,516 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-05 21:49:23,531 INFO [SplitLogWorker-instance-2:45565] regionserver.SplitLogWorker(130): SplitLogWorker instance-2.c.gcp-hbase.internal,45565,1515188960705 starting 2018-01-05 21:49:23,537 INFO [RS:0;instance-2:45565] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-05 21:49:23,558 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(813): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-05 21:49:23,562 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-05 21:49:23,604 DEBUG [ReplicationExecutor-0] zookeeper.ZKUtil(355): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,604 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$AdoptAbandonedQueuesWorker(788): Current list of replicators: [instance-2.c.gcp-hbase.internal,45565,1515188960705] other RSs: [instance-2.c.gcp-hbase.internal,45565,1515188960705] 2018-01-05 21:49:23,717 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=1, allServersCount=1 2018-01-05 21:49:23,728 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,735 INFO [RS:0;instance-2:45565] regionserver.HRegionServer(1536): Serving as instance-2.c.gcp-hbase.internal,45565,1515188960705, RpcServer on instance-2.c.gcp-hbase.internal/10.128.0.3:45565, sessionid=0x160c84cad8a0001 2018-01-05 21:49:23,742 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-05 21:49:23,742 DEBUG [RS:0;instance-2:45565] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,742 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'instance-2.c.gcp-hbase.internal,45565,1515188960705' 2018-01-05 21:49:23,742 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 21:49:23,750 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 21:49:23,752 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-05 21:49:23,752 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-05 21:49:23,752 DEBUG [RS:0;instance-2:45565] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:23,752 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'instance-2.c.gcp-hbase.internal,45565,1515188960705' 2018-01-05 21:49:23,752 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-05 21:49:23,753 DEBUG [RS:0;instance-2:45565] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-05 21:49:23,754 DEBUG [RS:0;instance-2:45565] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-05 21:49:23,754 INFO [RS:0;instance-2:45565] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-05 21:49:23,754 INFO [RS:0;instance-2:45565] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-05 21:49:24,825 INFO [RS:0;instance-2:45565] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=instance-2.c.gcp-hbase.internal%2C45565%2C1515188960705, suffix=, logDir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705, archiveDir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/oldWALs 2018-01-05 21:49:24,898 DEBUG [RS:0;instance-2:45565] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(417): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-01-05 21:49:24,985 DEBUG [RS:0;instance-2:45565] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(490): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:488) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:552) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:628) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:504) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:69) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:44) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:55) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:244) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2123) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1315) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1008) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:745) 2018-01-05 21:49:24,989 DEBUG [RS:0;instance-2:45565] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(524): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-01-05 21:49:25,037 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:657) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:117) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:716) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:711) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:745) 2018-01-05 21:49:25,050 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:51755,DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71,DISK] 2018-01-05 21:49:25,113 INFO [RS:0;instance-2:45565] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705/instance-2.c.gcp-hbase.internal%2C45565%2C1515188960705.1515188964874 2018-01-05 21:49:25,114 DEBUG [RS:0;instance-2:45565] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:51755,DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71,DISK]] 2018-01-05 21:49:25,151 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:25,167 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-05 21:49:25,171 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(213): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:25,323 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:25,362 DEBUG [RSProcedureDispatcher-pool4-t1] master.ServerManager(783): New admin connection to instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:25,366 DEBUG [RSProcedureDispatcher-pool4-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:45565 2018-01-05 21:49:25,372 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 21:49:25,374 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 51926 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:25,391 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open hbase:meta,,1.1588230740 2018-01-05 21:49:25,399 INFO [RS_OPEN_META-instance-2:45565-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-05 21:49:25,406 INFO [RS_OPEN_META-instance-2:45565-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=instance-2.c.gcp-hbase.internal%2C45565%2C1515188960705.meta, suffix=.meta, logDir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705, archiveDir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/oldWALs 2018-01-05 21:49:25,431 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:51755,DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3,DISK] 2018-01-05 21:49:25,487 INFO [RS_OPEN_META-instance-2:45565-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705/instance-2.c.gcp-hbase.internal%2C45565%2C1515188960705.meta.1515188965411.meta 2018-01-05 21:49:25,487 DEBUG [RS_OPEN_META-instance-2:45565-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:51755,DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3,DISK]] 2018-01-05 21:49:25,488 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-05 21:49:25,528 DEBUG [RS_OPEN_META-instance-2:45565-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-05 21:49:25,560 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-05 21:49:25,574 INFO [RS_OPEN_META-instance-2:45565-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-05 21:49:25,590 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-05 21:49:25,591 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-05 21:49:25,607 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/info 2018-01-05 21:49:25,608 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/info 2018-01-05 21:49:25,608 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:25,609 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:25,612 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:25,617 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_barrier 2018-01-05 21:49:25,617 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_barrier 2018-01-05 21:49:25,618 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:25,618 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:25,619 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:25,624 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_meta 2018-01-05 21:49:25,624 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_meta 2018-01-05 21:49:25,625 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:25,625 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:25,626 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:25,634 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_position 2018-01-05 21:49:25,634 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/rep_position 2018-01-05 21:49:25,635 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:25,635 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:25,638 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:25,642 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/table 2018-01-05 21:49:25,643 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/table 2018-01-05 21:49:25,643 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:25,643 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:25,644 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:25,650 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740 2018-01-05 21:49:25,660 DEBUG [RS_OPEN_META-instance-2:45565-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-05 21:49:25,673 DEBUG [RS_OPEN_META-instance-2:45565-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-05 21:49:25,676 INFO [RS_OPEN_META-instance-2:45565-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-05 21:49:25,726 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2205): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-05 21:49:25,749 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:25,759 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2229): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-05 21:49:25,768 DEBUG [RS_OPEN_META-instance-2:45565-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:25,855 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188965758}]},"ts":1515188965758} 2018-01-05 21:49:25,944 DEBUG [hconnection-0x3769ac70-shared-pool6-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:45565 2018-01-05 21:49:25,959 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 21:49:25,960 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 51929 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:26,028 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-05 21:49:26,033 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:26,037 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-05 21:49:26,042 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-05 21:49:26,047 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-05 21:49:26,047 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 2.5300sec 2018-01-05 21:49:26,155 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 2.9510sec 2018-01-05 21:49:26,157 INFO [instance-2:37786.masterManager] master.HMaster(3516): hbase:meta (default replica) deployed at=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:26,195 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.AssignmentManager(996): META REPORTED: rit=OPEN, location=instance-2.c.gcp-hbase.internal,45565,1515188960705, table=hbase:meta, region=1588230740 2018-01-05 21:49:26,197 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.AssignmentManager(998): META REPORTED but no procedure found (complete?); set location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:26,270 INFO [instance-2:37786.masterManager] master.MasterWalManager(189): Log folder hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/WALs/instance-2.c.gcp-hbase.internal,45565,1515188960705 belongs to an existing region server 2018-01-05 21:49:26,271 INFO [instance-2:37786.masterManager] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-05 21:49:26,288 INFO [instance-2:37786.masterManager] assignment.AssignmentManager(1212): Number of RegionServers=1 2018-01-05 21:49:26,288 INFO [instance-2:37786.masterManager] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515188971288 2018-01-05 21:49:26,290 INFO [instance-2:37786.masterManager] assignment.AssignmentManager(1219): Joined the cluster in 17msec, failover=true 2018-01-05 21:49:26,334 INFO [instance-2:37786.masterManager] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-05 21:49:26,339 INFO [instance-2:37786.masterManager] master.HMaster(1765): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-05 21:49:26,465 DEBUG [instance-2:37786.masterManager] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:26,466 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:26,487 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:26,530 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:26,542 WARN [DataStreamer for file /user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp/data/hbase/namespace/.tmp/.tableinfo.0000000001] hdfs.DFSOutputStream$DataStreamer(718): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1245) at java.lang.Thread.join(Thread.java:1319) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:716) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:476) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:652) 2018-01-05 21:49:26,546 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-05 21:49:26,556 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp Table name == hbase:namespace 2018-01-05 21:49:26,606 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741833_1009{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 42 2018-01-05 21:49:27,009 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,010 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76.: disabling compactions & flushes 2018-01-05 21:49:27,010 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,010 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,030 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:27,054 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 21:49:27,071 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 21:49:27,080 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:27,080 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188967080}]},"ts":1515188967080} 2018-01-05 21:49:27,100 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-05 21:49:27,101 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705}] 2018-01-05 21:49:27,108 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(813): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 hbase:namespace hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,116 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705; forceNewPlan=false, retain=false 2018-01-05 21:49:27,269 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,323 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x37e1d6a4 to localhost:50569, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 21:49:27,342 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22bf846a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 21:49:27,349 DEBUG [MultiHConnection-shared--pool7-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:45565 2018-01-05 21:49:27,357 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 21:49:27,358 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 51945 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:27,370 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(213): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,520 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:27,532 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,545 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] regionserver.HRegion(6949): Opening region: {ENCODED => 10be69381c2577a69f94b2de09731c76, NAME => 'hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76.', STARTKEY => '', ENDKEY => ''} 2018-01-05 21:49:27,554 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 10be69381c2577a69f94b2de09731c76 2018-01-05 21:49:27,558 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,590 DEBUG [StoreOpener-10be69381c2577a69f94b2de09731c76-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/namespace/10be69381c2577a69f94b2de09731c76/info 2018-01-05 21:49:27,591 DEBUG [StoreOpener-10be69381c2577a69f94b2de09731c76-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/namespace/10be69381c2577a69f94b2de09731c76/info 2018-01-05 21:49:27,592 INFO [StoreOpener-10be69381c2577a69f94b2de09731c76-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 21:49:27,592 INFO [StoreOpener-10be69381c2577a69f94b2de09731c76-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:27,594 INFO [StoreOpener-10be69381c2577a69f94b2de09731c76-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:27,601 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/namespace/10be69381c2577a69f94b2de09731c76 2018-01-05 21:49:27,619 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/hbase/namespace/10be69381c2577a69f94b2de09731c76/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 21:49:27,619 INFO [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] regionserver.HRegion(960): Onlined 10be69381c2577a69f94b2de09731c76; next sequenceid=2 2018-01-05 21:49:27,623 INFO [PostOpenDeployTasks:10be69381c2577a69f94b2de09731c76] regionserver.HRegionServer(2205): Post open deploy tasks for hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,627 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,628 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,632 DEBUG [PostOpenDeployTasks:10be69381c2577a69f94b2de09731c76] regionserver.HRegionServer(2229): Finished post open deploy task for hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. 2018-01-05 21:49:27,632 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:45565-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515188966335.10be69381c2577a69f94b2de09731c76. on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,654 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:27,667 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-05 21:49:27,667 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:27,668 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=10be69381c2577a69f94b2de09731c76, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 545msec 2018-01-05 21:49:27,669 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188967668}]},"ts":1515188967668} 2018-01-05 21:49:27,684 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-05 21:49:27,695 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:27,701 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3540sec 2018-01-05 21:49:27,728 DEBUG [instance-2:37786.masterManager] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-05 21:49:27,732 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-05 21:49:27,909 DEBUG [instance-2:37786.masterManager] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,910 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,921 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,928 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,955 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,959 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-05 21:49:27,963 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-05 21:49:27,981 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:27,987 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 189msec 2018-01-05 21:49:28,105 DEBUG [instance-2:37786.masterManager] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,106 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,115 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,121 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,131 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,133 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-05 21:49:28,144 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-05 21:49:28,146 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:49:28,146 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-05 21:49:28,151 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 149msec 2018-01-05 21:49:28,169 DEBUG [instance-2:37786.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/namespace/default already exists 2018-01-05 21:49:28,171 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-05 21:49:28,173 DEBUG [instance-2:37786.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/namespace/hbase already exists 2018-01-05 21:49:28,175 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-05 21:49:28,176 INFO [instance-2:37786.masterManager] master.HMaster(906): Master has completed initialization 7.532sec 2018-01-05 21:49:28,190 INFO [instance-2:37786.masterManager] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-05 21:49:28,190 INFO [instance-2:37786.masterManager] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-05 21:49:28,264 INFO [main] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x7a3e5cd3 to localhost:50569, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 21:49:28,288 DEBUG [main] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c79088e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 21:49:28,319 DEBUG [hconnection-0x2538bc06-shared-pool11-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:45565 2018-01-05 21:49:28,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 21:49:28,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 51950 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:28,381 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up 2018-01-05 21:49:28,470 INFO [main] hbase.ResourceChecker(148): before: client.TestMultiRespectsLimits#testBlockMultiLimits Thread=211, OpenFileDescriptor=422, MaxFileDescriptor=4096, SystemLoadAverage=1141, ProcessCount=95, AvailableMemoryMB=2452 2018-01-05 21:49:28,485 DEBUG [main] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:37786 2018-01-05 21:49:28,502 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 21:49:28,504 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 41667 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "5ce93511290d2bcf69e41d7ca5cec7464076d81d" user: "jenkins" date: "Fri Jan 5 21:18:13 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 21:49:28,540 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.HMaster$3(1732): Client=jenkins//10.128.0.3 create 'testBlockMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-05 21:49:28,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:28,654 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:28,681 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 21:49:28,739 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 492 2018-01-05 21:49:28,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 21:49:29,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 21:49:29,146 DEBUG [ProcExecWrkr-14] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp/data/default/testBlockMultiLimits/.tabledesc/.tableinfo.0000000001 2018-01-05 21:49:29,157 INFO [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(6772): creating HRegion testBlockMultiLimits HTD == 'testBlockMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp Table name == testBlockMultiLimits 2018-01-05 21:49:29,231 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|FINALIZED]]} size 0 2018-01-05 21:49:29,235 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(807): Instantiated testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,239 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1481): Closing testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2.: disabling compactions & flushes 2018-01-05 21:49:29,239 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1520): Updates disabled for region testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,239 INFO [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1633): Closed testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,256 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:29,260 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 21:49:29,276 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 21:49:29,281 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:29,282 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testBlockMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188969281}]},"ts":1515188969281} 2018-01-05 21:49:29,297 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1786): Updated table testBlockMultiLimits state to ENABLING in META 2018-01-05 21:49:29,297 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705}] 2018-01-05 21:49:29,304 INFO [ProcExecWrkr-14] procedure.MasterProcedureScheduler(813): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 testBlockMultiLimits testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,308 INFO [ProcExecWrkr-14] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705; forceNewPlan=false, retain=false 2018-01-05 21:49:29,324 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 21:49:29,324 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 21:49:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 21:49:29,461 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,483 INFO [ProcExecWrkr-16] assignment.RegionTransitionProcedure(213): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,633 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:29,644 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,661 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(6949): Opening region: {ENCODED => 266f0ed6152316e25529d37a971319b2, NAME => 'testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2.', STARTKEY => '', ENDKEY => ''} 2018-01-05 21:49:29,662 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testBlockMultiLimits 266f0ed6152316e25529d37a971319b2 2018-01-05 21:49:29,662 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(807): Instantiated testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,712 DEBUG [StoreOpener-266f0ed6152316e25529d37a971319b2-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/D 2018-01-05 21:49:29,712 DEBUG [StoreOpener-266f0ed6152316e25529d37a971319b2-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/D 2018-01-05 21:49:29,724 INFO [StoreOpener-266f0ed6152316e25529d37a971319b2-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 21:49:29,725 INFO [StoreOpener-266f0ed6152316e25529d37a971319b2-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 21:49:29,725 INFO [StoreOpener-266f0ed6152316e25529d37a971319b2-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:29,730 INFO [StoreOpener-266f0ed6152316e25529d37a971319b2-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:29,732 INFO [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,735 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2 2018-01-05 21:49:29,741 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testBlockMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 21:49:29,755 DEBUG [RS_OPEN_REGION-instance-2:45565-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 21:49:29,755 INFO [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(960): Onlined 266f0ed6152316e25529d37a971319b2; next sequenceid=2 2018-01-05 21:49:29,763 INFO [PostOpenDeployTasks:266f0ed6152316e25529d37a971319b2] regionserver.HRegionServer(2205): Post open deploy tasks for testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,768 INFO [ProcExecWrkr-1] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,771 DEBUG [PostOpenDeployTasks:266f0ed6152316e25529d37a971319b2] regionserver.HRegionServer(2229): Finished post open deploy task for testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:29,771 DEBUG [RS_OPEN_REGION-instance-2:45565-0] handler.OpenRegionHandler(128): Opened testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,790 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:29,794 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testBlockMultiLimits; resume parent processing. 2018-01-05 21:49:29,795 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testBlockMultiLimits, region=266f0ed6152316e25529d37a971319b2, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 487msec 2018-01-05 21:49:29,795 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:29,795 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testBlockMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188969795}]},"ts":1515188969795} 2018-01-05 21:49:29,806 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1786): Updated table testBlockMultiLimits state to ENABLED in META 2018-01-05 21:49:29,811 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 21:49:29,815 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; CreateTableProcedure table=testBlockMultiLimits in 1.2710sec 2018-01-05 21:49:29,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 21:49:29,844 INFO [main] client.HBaseAdmin$TableFuture(3632): Operation: CREATE, Table Name: default:testBlockMultiLimits completed 2018-01-05 21:49:30,064 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(849): Client=jenkins//10.128.0.3 procedure request for: flush-table-proc 2018-01-05 21:49:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] procedure.ProcedureCoordinator(177): Submitting procedure testBlockMultiLimits 2018-01-05 21:49:30,117 INFO [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(196): Starting procedure 'testBlockMultiLimits' 2018-01-05 21:49:30,121 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2018-01-05 21:49:30,122 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(204): Procedure 'testBlockMultiLimits' starting 'acquire' 2018-01-05 21:49:30,122 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(247): Starting procedure 'testBlockMultiLimits', kicking off acquire phase on members. 2018-01-05 21:49:30,127 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,128 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(93): Creating acquire znode:/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 21:49:30,130 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-05 21:49:30,130 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(101): Watching for acquire node:/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,130 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(105): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2018-01-05 21:49:30,131 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 21:49:30,132 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(183): Found procedure znode: /hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 21:49:30,133 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,133 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,134 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2018-01-05 21:49:30,134 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(209): start proc data length is 4 2018-01-05 21:49:30,134 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(211): Found data for znode:/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 21:49:30,134 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(154): Launching subprocedure to flush regions for testBlockMultiLimits 2018-01-05 21:49:30,156 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testBlockMultiLimits 2018-01-05 21:49:30,166 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(157): Starting subprocedure 'testBlockMultiLimits' with timeout 60000ms 2018-01-05 21:49:30,167 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2018-01-05 21:49:30,171 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(165): Subprocedure 'testBlockMultiLimits' starting 'acquire' stage 2018-01-05 21:49:30,184 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2018-01-05 21:49:30,187 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(245): Waiting for local region flush to finish. 2018-01-05 21:49:30,186 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:30,192 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. started... 2018-01-05 21:49:30,195 INFO [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=2.50 KB 2018-01-05 21:49:30,251 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2.store: D 2018-01-05 21:49:30,255 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-01-05 21:49:30,626 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|FINALIZED]]} size 0 2018-01-05 21:49:30,639 INFO [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=10, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/.tmp/D/a7651b572aad4461ab14e4c7bedfcbe9 2018-01-05 21:49:30,866 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/.tmp/D/a7651b572aad4461ab14e4c7bedfcbe9 as hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/D/a7651b572aad4461ab14e4c7bedfcbe9 2018-01-05 21:49:30,887 INFO [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.HStore(1028): Added hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testBlockMultiLimits/266f0ed6152316e25529d37a971319b2/D/a7651b572aad4461ab14e4c7bedfcbe9, entries=6, sequenceid=10, filesize=5.2 K 2018-01-05 21:49:30,917 INFO [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] regionserver.HRegion(2712): Finished memstore flush of ~2.50 KB/2556, currentsize=0 B/0 for region testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. in 722ms, sequenceid=10, compaction requested=false 2018-01-05 21:49:30,919 DEBUG [rs(instance-2.c.gcp-hbase.internal,45565,1515188960705)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testBlockMultiLimits,,1515188968534.266f0ed6152316e25529d37a971319b2. 2018-01-05 21:49:30,920 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1/1 local region flush tasks. 2018-01-05 21:49:30,925 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(258): Completed 1 local region flush tasks. 2018-01-05 21:49:30,925 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(291): cancelling 0 flush region tasks instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,925 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(167): Subprocedure 'testBlockMultiLimits' locally acquired 2018-01-05 21:49:30,925 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(240): Member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' joining acquired barrier for procedure (testBlockMultiLimits) in zk 2018-01-05 21:49:30,936 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,936 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,936 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 21:49:30,936 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 21:49:30,937 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(248): Watch for global barrier reached:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,937 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 21:49:30,938 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 21:49:30,938 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] zookeeper.ZKUtil(357): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,938 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(172): Subprocedure 'testBlockMultiLimits' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2018-01-05 21:49:30,938 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 21:49:30,939 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,939 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 21:49:30,940 DEBUG [main-EventThread] procedure.Procedure(298): member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' joining acquired barrier for procedure 'testBlockMultiLimits' on coordinator 2018-01-05 21:49:30,940 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@2fa6d705[Count = 0] remaining members to acquire global barrier 2018-01-05 21:49:30,940 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(212): Procedure 'testBlockMultiLimits' starting 'in-barrier' execution. 2018-01-05 21:49:30,940 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(117): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,944 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,944 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(78): Received created event:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,944 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(129): Received reached global barrier:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,944 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(186): Subprocedure 'testBlockMultiLimits' received 'reached' from coordinator. 2018-01-05 21:49:30,945 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(188): Subprocedure 'testBlockMultiLimits' locally completed 2018-01-05 21:49:30,945 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(264): Marking procedure 'testBlockMultiLimits' completed for member 'instance-2.c.gcp-hbase.internal,45565,1515188960705' in zk 2018-01-05 21:49:30,946 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,946 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2018-01-05 21:49:30,949 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,949 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,949 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 21:49:30,949 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 21:49:30,949 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(193): Subprocedure 'testBlockMultiLimits' has notified controller of completion 2018-01-05 21:49:30,949 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2018-01-05 21:49:30,949 DEBUG [member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1] procedure.Subprocedure(218): Subprocedure 'testBlockMultiLimits' completed. 2018-01-05 21:49:30,950 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 21:49:30,950 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 21:49:30,951 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 21:49:30,951 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,952 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 21:49:30,952 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 21:49:30,952 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,953 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(221): Finished data from procedure 'testBlockMultiLimits' member 'instance-2.c.gcp-hbase.internal,45565,1515188960705': 2018-01-05 21:49:30,953 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' released barrier for procedure'testBlockMultiLimits', counting down latch. Waiting for 0 more 2018-01-05 21:49:30,954 INFO [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(221): Procedure 'testBlockMultiLimits' execution completed 2018-01-05 21:49:30,954 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(230): Running finish phase. 2018-01-05 21:49:30,954 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2018-01-05 21:49:30,954 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(164): Attempting to clean out zk node for op:testBlockMultiLimits 2018-01-05 21:49:30,954 INFO [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureUtil(286): Clearing all znodes for procedure testBlockMultiLimitsincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-05 21:49:30,958 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,958 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(78): Received created event:/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,958 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,959 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-05 21:49:30,959 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(108): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2018-01-05 21:49:30,959 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 21:49:30,959 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,959 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,959 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,960 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 21:49:30,960 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 21:49:30,961 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,961 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 21:49:30,962 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(355): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,962 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 21:49:30,963 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 21:49:30,964 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 21:49:30,968 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,982 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-05 21:49:30,982 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(105): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2018-01-05 21:49:30,982 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 21:49:30,985 DEBUG [(instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2018-01-05 21:49:30,985 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 21:49:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] flush.MasterFlushTableProcedureManager(168): Done waiting - exec procedure flush-table-proc for 'testBlockMultiLimits' 2018-01-05 21:49:30,986 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] flush.MasterFlushTableProcedureManager(170): Master flush table procedure is successful! 2018-01-05 21:49:30,986 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:45565-0x160c84cad8a0001, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-05 21:49:30,986 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(108): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2018-01-05 21:49:30,986 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 21:49:30,987 DEBUG [main-EventThread] zookeeper.ZKUtil(614): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testBlockMultiLimits because node does not exist (not an error) 2018-01-05 21:49:30,987 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,987 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 21:49:30,988 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 21:49:30,988 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:30,988 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,988 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 21:49:30,988 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:37786-0x160c84cad8a0000, quorum=localhost:50569, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 21:49:30,991 DEBUG [main] client.HBaseAdmin(2777): Waiting a max of 300000 ms for procedure 'flush-table-proc : testBlockMultiLimits'' to complete. (max 20000 ms per retry) 2018-01-05 21:49:30,992 DEBUG [main] client.HBaseAdmin(2786): (#1) Sleeping: 100ms while waiting for procedure completion. 2018-01-05 21:49:31,093 DEBUG [main] client.HBaseAdmin(2792): Getting current status of procedure from master... 2018-01-05 21:49:31,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1109): Checking to see if procedure from request:flush-table-proc is done 2018-01-05 21:49:31,144 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 21:49:31,146 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 21:49:31,592 INFO [main] hbase.ResourceChecker(172): after: client.TestMultiRespectsLimits#testBlockMultiLimits Thread=217 (was 211) Potentially hanging thread: hconnection-0x3769ac70-shared-pool6-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2538bc06-metaLookup-shared--pool12-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2538bc06-shared-pool11-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_697808235_1 at /127.0.0.1:60399 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3769ac70-shared-pool6-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x3769ac70-shared-pool6-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MultiHConnection-shared--pool7-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MultiHConnection-shared--pool7-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2538bc06-metaLookup-shared--pool12-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3769ac70-shared-pool6-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2538bc06-shared-pool11-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Default-IPC-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3769ac70-shared-pool6-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (instance-2.c.gcp-hbase.internal,37786,1515188957846)-proc-coordinator-pool2-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_REGION-instance-2:45565-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RSProcedureDispatcher-pool4-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'instance-2.c.gcp-hbase.internal,45565,1515188960705' subprocedure-pool1-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=427 (was 422) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=1105 (was 1141), ProcessCount=96 (was 95) - ProcessCount LEAK? -, AvailableMemoryMB=2426 (was 2452) 2018-01-05 21:49:31,640 INFO [main] hbase.ResourceChecker(148): before: client.TestMultiRespectsLimits#testMultiLimits Thread=217, OpenFileDescriptor=427, MaxFileDescriptor=4096, SystemLoadAverage=1105, ProcessCount=95, AvailableMemoryMB=2427 2018-01-05 21:49:31,664 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.HMaster$3(1732): Client=jenkins//10.128.0.3 create 'testMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-05 21:49:31,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] procedure2.ProcedureExecutor(869): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:31,775 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 21:49:31,790 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:31,854 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:31,863 DEBUG [ProcExecWrkr-3] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp/data/default/testMultiLimits/.tabledesc/.tableinfo.0000000001 2018-01-05 21:49:31,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 21:49:31,887 INFO [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(6772): creating HRegion testMultiLimits HTD == 'testMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/.tmp Table name == testMultiLimits 2018-01-05 21:49:31,973 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|FINALIZED]]} size 0 2018-01-05 21:49:31,978 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(807): Instantiated testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:31,980 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1481): Closing testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.: disabling compactions & flushes 2018-01-05 21:49:31,980 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:31,980 INFO [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1633): Closed testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,007 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:32,008 DEBUG [ProcExecWrkr-3] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 21:49:32,019 INFO [ProcExecWrkr-3] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 21:49:32,024 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:32,025 DEBUG [ProcExecWrkr-3] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188972025}]},"ts":1515188972025} 2018-01-05 21:49:32,038 INFO [ProcExecWrkr-3] hbase.MetaTableAccessor(1786): Updated table testMultiLimits state to ENABLING in META 2018-01-05 21:49:32,039 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705}] 2018-01-05 21:49:32,043 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(813): pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 testMultiLimits testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,051 INFO [ProcExecWrkr-3] assignment.AssignProcedure(211): Start pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705; forceNewPlan=false, retain=false 2018-01-05 21:49:32,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 21:49:32,204 INFO [ProcExecWrkr-5] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,244 INFO [ProcExecWrkr-5] assignment.RegionTransitionProcedure(213): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,396 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:32,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 21:49:32,407 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,425 DEBUG [RS_OPEN_REGION-instance-2:45565-1] regionserver.HRegion(6949): Opening region: {ENCODED => 3031467ed386262b1b794af9aceadb44, NAME => 'testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.', STARTKEY => '', ENDKEY => ''} 2018-01-05 21:49:32,426 DEBUG [RS_OPEN_REGION-instance-2:45565-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 3031467ed386262b1b794af9aceadb44 2018-01-05 21:49:32,426 DEBUG [RS_OPEN_REGION-instance-2:45565-1] regionserver.HRegion(807): Instantiated testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,451 DEBUG [StoreOpener-3031467ed386262b1b794af9aceadb44-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D 2018-01-05 21:49:32,451 DEBUG [StoreOpener-3031467ed386262b1b794af9aceadb44-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D 2018-01-05 21:49:32,452 INFO [StoreOpener-3031467ed386262b1b794af9aceadb44-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 21:49:32,457 INFO [StoreOpener-3031467ed386262b1b794af9aceadb44-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 21:49:32,459 INFO [StoreOpener-3031467ed386262b1b794af9aceadb44-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:32,460 INFO [StoreOpener-3031467ed386262b1b794af9aceadb44-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:32,463 INFO [RS_OPEN_REGION-instance-2:45565-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,465 DEBUG [RS_OPEN_REGION-instance-2:45565-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44 2018-01-05 21:49:32,470 DEBUG [RS_OPEN_REGION-instance-2:45565-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 21:49:32,499 DEBUG [RS_OPEN_REGION-instance-2:45565-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 21:49:32,500 INFO [RS_OPEN_REGION-instance-2:45565-1] regionserver.HRegion(960): Onlined 3031467ed386262b1b794af9aceadb44; next sequenceid=2 2018-01-05 21:49:32,507 INFO [PostOpenDeployTasks:3031467ed386262b1b794af9aceadb44] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,515 INFO [ProcExecWrkr-6] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,520 DEBUG [PostOpenDeployTasks:3031467ed386262b1b794af9aceadb44] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:32,520 DEBUG [RS_OPEN_REGION-instance-2:45565-1] handler.OpenRegionHandler(128): Opened testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,544 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:32,550 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMultiLimits; resume parent processing. 2018-01-05 21:49:32,550 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 501msec 2018-01-05 21:49:32,551 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:32,551 DEBUG [ProcExecWrkr-6] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515188972551}]},"ts":1515188972551} 2018-01-05 21:49:32,582 INFO [ProcExecWrkr-6] hbase.MetaTableAccessor(1786): Updated table testMultiLimits state to ENABLED in META 2018-01-05 21:49:32,588 DEBUG [ProcExecWrkr-6] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 21:49:32,592 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMultiLimits in 923msec 2018-01-05 21:49:32,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 21:49:32,908 INFO [main] client.HBaseAdmin$TableFuture(3632): Operation: CREATE, Table Name: default:testMultiLimits completed 2018-01-05 21:49:32,909 DEBUG [main] hbase.HBaseTestingUtility(3262): Waiting until all regions of table testMultiLimits get assigned. Timeout = 60000ms 2018-01-05 21:49:32,910 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 21:49:32,943 INFO [main] hbase.HBaseTestingUtility(3314): All regions for table testMultiLimits assigned to meta. Checking AM states. 2018-01-05 21:49:32,944 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 21:49:32,944 INFO [main] hbase.HBaseTestingUtility(3334): All regions for table testMultiLimits assigned. 2018-01-05 21:49:33,712 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices$1(1125): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 17576 Client: jenkins//10.128.0.3 first region in multi=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:34,415 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.HRegion(8152): writing data to region testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. with WAL disabled. Data may be lost in the event of a crash. 2018-01-05 21:49:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] master.HMaster$2(1605): Client=jenkins//10.128.0.3 split testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:35,133 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.SplitTableRegionProcedure(141): splitKey isn't explicitly specified, will try to find a best split key from RS 2018-01-05 21:49:35,181 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=480.59 KB 2018-01-05 21:49:35,182 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.store: D 2018-01-05 21:49:35,183 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-01-05 21:49:35,334 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 21:49:35,335 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 21:49:35,510 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:35,513 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=5, memsize=480.6 K, hasBloomFilter=false, into tmp file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/.tmp/D/424eb00a97ef4d9d85f24c8fdb4f7f04 2018-01-05 21:49:35,562 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/.tmp/D/424eb00a97ef4d9d85f24c8fdb4f7f04 as hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04 2018-01-05 21:49:35,577 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.HStore(1028): Added hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04, entries=17576, sequenceid=5, filesize=502.8 K 2018-01-05 21:49:35,580 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45565] regionserver.HRegion(2712): Finished memstore flush of ~480.59 KB/492128, currentsize=0 B/0 for region testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. in 399ms, sequenceid=5, compaction requested=false 2018-01-05 21:49:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.SplitTableRegionProcedure(153): Splittable=true rit=OPEN, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:35,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] procedure2.ProcedureExecutor(869): Stored pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:35,786 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(813): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 testMultiLimits testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:35,787 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:35,792 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 21:49:35,830 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:35,834 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:35,835 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705}] 2018-01-05 21:49:35,896 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(813): pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705 testMultiLimits testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:35,897 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44., regionState=CLOSING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:35,913 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(213): Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=CLOSING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:36,064 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:36,114 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1531): Close 3031467ed386262b1b794af9aceadb44 without moving 2018-01-05 21:49:36,120 DEBUG [RS_CLOSE_REGION-instance-2:45565-0] handler.CloseRegionHandler(92): Processing close of testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:36,121 DEBUG [RS_CLOSE_REGION-instance-2:45565-0] regionserver.HRegion(1481): Closing testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.: disabling compactions & flushes 2018-01-05 21:49:36,121 DEBUG [RS_CLOSE_REGION-instance-2:45565-0] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:36,134 INFO [StoreCloserThread-testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.-1] regionserver.HStore(930): Closed D 2018-01-05 21:49:36,152 DEBUG [RS_CLOSE_REGION-instance-2:45565-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2018-01-05 21:49:36,187 INFO [RS_CLOSE_REGION-instance-2:45565-0] regionserver.HRegion(1633): Closed testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:36,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=CLOSING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:36,202 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44., regionState=CLOSED 2018-01-05 21:49:36,202 DEBUG [RS_CLOSE_REGION-instance-2:45565-0] handler.CloseRegionHandler(125): Closed testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44. 2018-01-05 21:49:36,239 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=12, ppid=11, state=SUCCESS; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:36,244 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895; resume parent processing. 2018-01-05 21:49:36,244 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=12, ppid=11, state=SUCCESS; UnassignProcedure table=testMultiLimits, region=3031467ed386262b1b794af9aceadb44, server=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 399msec 2018-01-05 21:49:36,244 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:36,285 INFO [ProcExecWrkr-11] assignment.SplitTableRegionProcedure(587): pid=11 splitting 1 storefiles, region=3031467ed386262b1b794af9aceadb44, threads=1 2018-01-05 21:49:36,285 INFO [ProcExecWrkr-11] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:36,296 DEBUG [StoreFileSplitter-%1$d-pool13-t1] assignment.SplitTableRegionProcedure(667): pid=11 splitting started for store file: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04 for region: 3031467ed386262b1b794af9aceadb44 2018-01-05 21:49:36,444 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:36,556 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:36,561 DEBUG [StoreFileSplitter-%1$d-pool13-t1] assignment.SplitTableRegionProcedure(678): pid=11 splitting complete for store file: hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04 for region: 3031467ed386262b1b794af9aceadb44 2018-01-05 21:49:36,561 DEBUG [ProcExecWrkr-11] assignment.SplitTableRegionProcedure(648): pid=11 split storefiles for region 3031467ed386262b1b794af9aceadb44 Daughter A: 1 storefiles, Daughter B: 1 storefiles. 2018-01-05 21:49:36,642 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741842_1018{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 53 2018-01-05 21:49:37,121 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:37,134 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:37,138 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:37,140 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":3,"row":"testMultiLimits,,1515188971664.3031467ed386262b1b794af9aceadb44.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":1515188977139},{"qualifier":"splitA","vlen":52,"tag":[],"timestamp":1515188977139},{"qualifier":"splitB","vlen":52,"tag":[],"timestamp":1515188977139}]},"ts":1515188977139} 2018-01-05 21:49:37,140 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":1515188977139},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":1515188977139}]},"ts":1515188977139} 2018-01-05 21:49:37,140 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":1515188977139},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":1515188977139}]},"ts":1515188977139} 2018-01-05 21:49:37,853 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:37,858 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:37,858 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705}, {pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705}] 2018-01-05 21:49:37,864 INFO [ProcExecWrkr-11] procedure.MasterProcedureScheduler(813): pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 testMultiLimits testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:37,865 INFO [ProcExecWrkr-12] procedure.MasterProcedureScheduler(813): pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 testMultiLimits testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:37,883 INFO [ProcExecWrkr-11] assignment.AssignProcedure(211): Start pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705; forceNewPlan=false, retain=false 2018-01-05 21:49:37,885 INFO [ProcExecWrkr-12] assignment.AssignProcedure(211): Start pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,45565,1515188960705; forceNewPlan=false, retain=false 2018-01-05 21:49:37,929 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 21:49:38,037 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,037 INFO [ProcExecWrkr-15] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,051 INFO [ProcExecWrkr-15] assignment.RegionTransitionProcedure(213): Dispatch pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,058 INFO [ProcExecWrkr-14] assignment.RegionTransitionProcedure(213): Dispatch pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,149 DEBUG [hconnection-0x2538bc06-shared-pool11-t47] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=0 of 16 failed; retrying after sleep of 100 because: No server address listed in hbase:meta for region testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. containing row aaa 2018-01-05 21:49:38,201 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,45565,1515188960705 version=3145728 2018-01-05 21:49:38,209 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,223 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=45565] regionserver.RSRpcServices(1906): Open testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,224 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(6949): Opening region: {ENCODED => 4b559548937a24f4fe9ff20b0930dad6, NAME => 'testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6.', STARTKEY => '', ENDKEY => 'njs'} 2018-01-05 21:49:38,225 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 4b559548937a24f4fe9ff20b0930dad6 2018-01-05 21:49:38,225 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(807): Instantiated testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,224 DEBUG [RS_OPEN_REGION-instance-2:45565-2] regionserver.HRegion(6949): Opening region: {ENCODED => 29ae2c7fbeae08e8252582990c4e7895, NAME => 'testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895.', STARTKEY => 'njs', ENDKEY => ''} 2018-01-05 21:49:38,229 DEBUG [RS_OPEN_REGION-instance-2:45565-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:38,230 DEBUG [RS_OPEN_REGION-instance-2:45565-2] regionserver.HRegion(807): Instantiated testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,258 DEBUG [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D 2018-01-05 21:49:38,258 DEBUG [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D 2018-01-05 21:49:38,266 INFO [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 21:49:38,271 INFO [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 21:49:38,270 DEBUG [hconnection-0x2538bc06-shared-pool11-t47] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=1 of 16 failed; retrying after sleep of 201 because: No server address listed in hbase:meta for region testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. containing row aaa 2018-01-05 21:49:38,271 INFO [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:38,272 INFO [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:38,281 DEBUG [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D 2018-01-05 21:49:38,283 DEBUG [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D 2018-01-05 21:49:38,283 INFO [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 21:49:38,283 INFO [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 21:49:38,284 INFO [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 21:49:38,284 INFO [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 21:49:38,369 DEBUG [StoreOpener-29ae2c7fbeae08e8252582990c4e7895-1] regionserver.HStore(549): loaded hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-top 2018-01-05 21:49:38,370 INFO [RS_OPEN_REGION-instance-2:45565-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,374 DEBUG [RS_OPEN_REGION-instance-2:45565-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:38,378 DEBUG [RS_OPEN_REGION-instance-2:45565-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 21:49:38,396 DEBUG [RS_OPEN_REGION-instance-2:45565-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/recovered.edits/7.seqid, newSeqId=7, maxSeqId=0 2018-01-05 21:49:38,396 INFO [RS_OPEN_REGION-instance-2:45565-2] regionserver.HRegion(960): Onlined 29ae2c7fbeae08e8252582990c4e7895; next sequenceid=7 2018-01-05 21:49:38,409 INFO [PostOpenDeployTasks:29ae2c7fbeae08e8252582990c4e7895] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,420 DEBUG [StoreOpener-4b559548937a24f4fe9ff20b0930dad6-1] regionserver.HStore(549): loaded hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-bottom 2018-01-05 21:49:38,420 INFO [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,422 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6 2018-01-05 21:49:38,425 DEBUG [RS_OPEN_REGION-instance-2:45565-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 21:49:38,441 DEBUG [PostOpenDeployTasks:29ae2c7fbeae08e8252582990c4e7895] regionserver.CompactSplit(350): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:0), split_queue=0 2018-01-05 21:49:38,463 DEBUG [RS_OPEN_REGION-instance-2:45565-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/recovered.edits/6.seqid, newSeqId=6, maxSeqId=0 2018-01-05 21:49:38,463 INFO [RS_OPEN_REGION-instance-2:45565-0] regionserver.HRegion(960): Onlined 4b559548937a24f4fe9ff20b0930dad6; next sequenceid=6 2018-01-05 21:49:38,474 INFO [PostOpenDeployTasks:4b559548937a24f4fe9ff20b0930dad6] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,483 DEBUG [RS:0;instance-2:45565-shortCompactions-1515188978429] compactions.SortedCompactionPolicy(68): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2018-01-05 21:49:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=7, pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,486 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895., regionState=OPEN, openSeqNum=7, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,487 DEBUG [PostOpenDeployTasks:29ae2c7fbeae08e8252582990c4e7895] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,497 DEBUG [PostOpenDeployTasks:4b559548937a24f4fe9ff20b0930dad6] regionserver.CompactSplit(350): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0 2018-01-05 21:49:38,500 DEBUG [RS_OPEN_REGION-instance-2:45565-2] handler.OpenRegionHandler(128): Opened testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37786] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=6, pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705; rit=OPENING, location=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,511 DEBUG [RS:0;instance-2:45565-longCompactions-1515188963331] compactions.SortedCompactionPolicy(68): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2018-01-05 21:49:38,511 INFO [ProcExecWrkr-2] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6., regionState=OPEN, openSeqNum=6, regionLocation=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,514 DEBUG [PostOpenDeployTasks:4b559548937a24f4fe9ff20b0930dad6] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,518 DEBUG [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.HStore(1722): 4b559548937a24f4fe9ff20b0930dad6 - D: Initiating minor compaction (all files) 2018-01-05 21:49:38,518 DEBUG [hconnection-0x2538bc06-shared-pool11-t47] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=2 of 16 failed; retrying after sleep of 300 because: No server address listed in hbase:meta for region testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. containing row aaa 2018-01-05 21:49:38,522 DEBUG [RS_OPEN_REGION-instance-2:45565-0] handler.OpenRegionHandler(128): Opened testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. on instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,526 DEBUG [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.HStore(1722): 29ae2c7fbeae08e8252582990c4e7895 - D: Initiating minor compaction (all files) 2018-01-05 21:49:38,527 INFO [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.HRegion(2066): Starting compaction on D in region testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6. 2018-01-05 21:49:38,528 INFO [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.HStore(1356): Starting compaction of [hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-bottom] into tmpdir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/.tmp, totalSize=502.8 K 2018-01-05 21:49:38,529 INFO [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.HRegion(2066): Starting compaction on D in region testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895. 2018-01-05 21:49:38,529 INFO [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.HStore(1356): Starting compaction of [hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-top] into tmpdir=hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/.tmp, totalSize=502.8 K 2018-01-05 21:49:38,532 DEBUG [RS:0;instance-2:45565-longCompactions-1515188963331] compactions.Compactor(204): Compacting hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-bottom, keycount=8788, bloomtype=NONE, size=502.8 K, encoding=NONE, seqNum=5, earliestPutTs=1515188974355 2018-01-05 21:49:38,535 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=14, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,541 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=14, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=29ae2c7fbeae08e8252582990c4e7895, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 669msec 2018-01-05 21:49:38,536 DEBUG [RS:0;instance-2:45565-shortCompactions-1515188978429] compactions.Compactor(204): Compacting hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D/424eb00a97ef4d9d85f24c8fdb4f7f04.3031467ed386262b1b794af9aceadb44-hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/3031467ed386262b1b794af9aceadb44/D/424eb00a97ef4d9d85f24c8fdb4f7f04-top, keycount=8788, bloomtype=NONE, size=502.8 K, encoding=NONE, seqNum=6, earliestPutTs=1515188974355 2018-01-05 21:49:38,535 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=13, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 2018-01-05 21:49:38,549 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895; resume parent processing. 2018-01-05 21:49:38,550 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=13, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=4b559548937a24f4fe9ff20b0930dad6, target=instance-2.c.gcp-hbase.internal,45565,1515188960705 in 672msec 2018-01-05 21:49:38,550 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 2018-01-05 21:49:38,556 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=11, state=SUCCESS; SplitTableRegionProcedure table=testMultiLimits, parent=3031467ed386262b1b794af9aceadb44, daughterA=4b559548937a24f4fe9ff20b0930dad6, daughterB=29ae2c7fbeae08e8252582990c4e7895 in 2.9300sec 2018-01-05 21:49:38,997 INFO [RS:0;instance-2:45565-shortCompactions-1515188978429] throttle.PressureAwareThroughputController(153): 29ae2c7fbeae08e8252582990c4e7895#D#compaction#2 average throughput is 0.59 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/sec 2018-01-05 21:49:39,011 INFO [RS:0;instance-2:45565-longCompactions-1515188963331] throttle.PressureAwareThroughputController(153): 4b559548937a24f4fe9ff20b0930dad6#D#compaction#3 average throughput is 0.61 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 10.00 MB/sec 2018-01-05 21:49:39,022 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cd143fe5-bd1b-4c27-a2b3-53ecba56aa71:NORMAL:127.0.0.1:51755|FINALIZED]]} size 0 2018-01-05 21:49:39,029 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51755 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-527defdd-0e3b-4ed9-9605-a76869a0b5f3:NORMAL:127.0.0.1:51755|RBW]]} size 0 2018-01-05 21:49:39,052 DEBUG [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/.tmp/D/c10117a7e0fc40e38f4bf9621c40aead as hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/29ae2c7fbeae08e8252582990c4e7895/D/c10117a7e0fc40e38f4bf9621c40aead 2018-01-05 21:49:39,104 DEBUG [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/.tmp/D/71e19011471e4c85a98756619edda07c as hdfs://localhost:42367/user/jenkins/test-data/a9a98117-a5f7-4de2-bf80-d4d2d9348ffc/data/default/testMultiLimits/4b559548937a24f4fe9ff20b0930dad6/D/71e19011471e4c85a98756619edda07c 2018-01-05 21:49:39,162 INFO [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.HStore(1486): Completed compaction of 1 (all) file(s) in D of 4b559548937a24f4fe9ff20b0930dad6 into 71e19011471e4c85a98756619edda07c(size=260.8 K), total size for store is 260.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2018-01-05 21:49:39,162 INFO [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.CompactSplit$CompactionRunner(594): Completed compaction: regionName = testMultiLimits,,1515188975587.4b559548937a24f4fe9ff20b0930dad6., storeName = D, priority = 15, time = 1515188978475; duration=0sec 2018-01-05 21:49:39,165 DEBUG [RS:0;instance-2:45565-longCompactions-1515188963331] regionserver.CompactSplit$CompactionRunner(622): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0 2018-01-05 21:49:39,169 INFO [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.HStore(1486): Completed compaction of 1 (all) file(s) in D of 29ae2c7fbeae08e8252582990c4e7895 into c10117a7e0fc40e38f4bf9621c40aead(size=246.5 K), total size for store is 246.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2018-01-05 21:49:39,175 INFO [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.CompactSplit$CompactionRunner(594): Completed compaction: regionName = testMultiLimits,njs,1515188975587.29ae2c7fbeae08e8252582990c4e7895., storeName = D, priority = 15, time = 1515188978429; duration=0sec 2018-01-05 21:49:39,176 DEBUG [RS:0;instance-2:45565-shortCompactions-1515188978429] regionserver.CompactSplit$CompactionRunner(622): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0 2018-01-05 21:49:43,851 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 21:49:43,853 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_1] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 21:49:43,853 INFO [instance-2.c.gcp-hbase.internal,37786,1515188957846_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 21:49:43,866 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 2015ms GC pool 'PS MarkSweep' had collection(s): count=1 time=1601ms GC pool 'PS Scavenge' had collection(s): count=1 time=648ms 2018-01-05 21:49:44,350 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 21:49:51,557 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 4689ms GC pool 'PS MarkSweep' had collection(s): count=1 time=3795ms GC pool 'PS Scavenge' had collection(s): count=1 time=1027ms 2018-01-05 21:49:51,560 INFO [instance-2.c.gcp-hbase.internal,37786,1515188957846_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 21:49:51,561 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 21:49:51,562 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 21:49:51,563 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 403 actions to finish on table: testMultiLimits 2018-01-05 21:49:52,750 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-05 21:49:52,750 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-05 21:49:52,750 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-05 21:50:01,279 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 21:50:01,280 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 21:50:01,286 INFO [instance-2.c.gcp-hbase.internal,37786,1515188957846_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 21:50:01,286 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 5727ms GC pool 'PS MarkSweep' had collection(s): count=1 time=4761ms GC pool 'PS Scavenge' had collection(s): count=1 time=970ms 2018-01-05 21:50:01,570 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 352 actions to finish on table: testMultiLimits 2018-01-05 21:50:17,117 WARN [RS:0;instance-2:45565] util.Sleeper(97): We slept 11812ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 21:50:17,119 INFO [instance-2.c.gcp-hbase.internal,37786,1515188957846_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 21:50:17,121 WARN [M:0;instance-2:37786] util.Sleeper(97): We slept 11836ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 21:50:17,123 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 21:50:17,123 INFO [instance-2.c.gcp-hbase.internal,45565,1515188960705_Chore_3] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 21:50:17,127 WARN [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(176): Detected pause in JVM or host machine (eg GC): pause of approximately 11339ms GC pool 'PS MarkSweep' had collection(s): count=1 time=10570ms GC pool 'PS Scavenge' had collection(s): count=1 time=1210ms 2018-01-05 21:50:17,127 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@75b21c3b] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 11342ms GC pool 'PS MarkSweep' had collection(s): count=1 time=10570ms GC pool 'PS Scavenge' had collection(s): count=1 time=1210ms 2018-01-05 21:50:17,128 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@2a492f2a] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 11342ms GC pool 'PS MarkSweep' had collection(s): count=1 time=10570ms GC pool 'PS Scavenge' had collection(s): count=1 time=1210ms 2018-01-05 21:50:17,128 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 305 actions to finish on table: testMultiLimits