2018-01-05 16:18:50,166 INFO [main] hbase.HBaseTestingUtility(945): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2018-01-05 16:18:51,230 INFO [main] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf, deleteOnExit=true 2018-01-05 16:18:51,233 INFO [main] hbase.HBaseTestingUtility(960): STARTING DFS 2018-01-05 16:18:51,245 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cache_data in system properties and HBase conf 2018-01-05 16:18:51,249 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/hadoop_tmp in system properties and HBase conf 2018-01-05 16:18:51,253 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/hadoop_logs in system properties and HBase conf 2018-01-05 16:18:51,256 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/mapred_local in system properties and HBase conf 2018-01-05 16:18:51,261 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/mapred_temp in system properties and HBase conf 2018-01-05 16:18:51,264 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2018-01-05 16:18:52,228 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-01-05 16:18:53,502 DEBUG [main] fs.HFileSystem(318): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-01-05 16:18:58,964 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-01-05 16:18:59,904 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-01-05 16:19:00,437 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-05 16:19:00,724 INFO [main] log.Slf4jLog(67): Extract jar:file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/%7BWORKSPACE%7D/maven-repo/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_38953_hdfs____.ms57b8/webapp 2018-01-05 16:19:01,569 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38953 2018-01-05 16:19:03,821 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-05 16:19:03,846 INFO [main] log.Slf4jLog(67): Extract jar:file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/%7BWORKSPACE%7D/maven-repo/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_53703_datanode____s3hy4j/webapp 2018-01-05 16:19:04,149 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53703 2018-01-05 16:19:07,035 ERROR [DataNode: [[[DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data1/, [DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44287] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-05 16:19:07,384 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1f730f6930cd98: from storage DS-d739aef8-699c-4dc0-8f0e-fadfec954349 node DatanodeRegistration(127.0.0.1:51821, datanodeUuid=cefd6a7c-2c7e-4d01-80b0-3bce99012e46, infoPort=56483, infoSecurePort=0, ipcPort=45229, storageInfo=lv=-56;cid=testClusterID;nsid=846693457;c=0), blocks: 0, hasStaleStorage: true, processing time: 3 msecs 2018-01-05 16:19:07,385 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1f730f6930cd98: from storage DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368 node DatanodeRegistration(127.0.0.1:51821, datanodeUuid=cefd6a7c-2c7e-4d01-80b0-3bce99012e46, infoPort=56483, infoSecurePort=0, ipcPort=45229, storageInfo=lv=-56;cid=testClusterID;nsid=846693457;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-05 16:19:07,748 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-05 16:19:07,814 INFO [main] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=55510 2018-01-05 16:19:07,858 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:07,863 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:09,538 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 7 2018-01-05 16:19:09,974 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995 with version=8 2018-01-05 16:19:09,976 INFO [main] hbase.HBaseTestingUtility(1204): Setting hbase.fs.tmp.dir to hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/hbase-staging 2018-01-05 16:19:12,617 INFO [main] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-01-05 16:19:14,626 INFO [main] client.ConnectionUtils(122): master/instance-2.c.gcp-hbase.internal/10.128.0.3:0 server-side Connection retries=45 2018-01-05 16:19:14,896 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-05 16:19:14,900 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-05 16:19:14,900 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-05 16:19:15,386 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-05 16:19:16,045 DEBUG [main] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-01-05 16:19:16,218 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/10.128.0.3:54757 2018-01-05 16:19:16,259 INFO [main] hfile.CacheConfig(553): Allocating On heap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-01-05 16:19:16,302 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:16,306 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:16,320 DEBUG [main] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-01-05 16:19:16,325 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:16,330 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:16,506 INFO [main] zookeeper.RecoverableZooKeeper(107): Process identifier=master:54757 connecting to ZooKeeper ensemble=localhost:55510 2018-01-05 16:19:16,663 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:547570x0, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 16:19:16,668 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:54757-0x160c71e6b450000 connected 2018-01-05 16:19:16,786 DEBUG [main] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-05 16:19:16,789 DEBUG [main] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 16:19:16,942 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=54757 2018-01-05 16:19:16,963 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=54757 2018-01-05 16:19:16,967 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=54757 2018-01-05 16:19:16,982 INFO [main] master.HMaster(476): hbase.rootdir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995, hbase.cluster.distributed=false 2018-01-05 16:19:17,066 INFO [main] master.HMaster(1987): Adding backup master ZNode /hbase/backup-masters/instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:17,086 DEBUG [main] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:17,260 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-05 16:19:17,271 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 16:19:17,272 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-05 16:19:17,275 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 16:19:17,277 INFO [instance-2:54757.masterManager] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/instance-2.c.gcp-hbase.internal,54757,1515169150919 from backup master directory 2018-01-05 16:19:17,290 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:17,293 WARN [instance-2:54757.masterManager] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-05 16:19:17,293 INFO [instance-2:54757.masterManager] master.ActiveMasterManager(181): Registered Active Master=instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:17,309 INFO [instance-2:54757.masterManager] regionserver.ChunkCreator(408): Allocating MemStoreChunkPool with chunk size 2 MB, max count 497, initial count 0 2018-01-05 16:19:17,390 INFO [main] client.ConnectionUtils(122): regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0 server-side Connection retries=45 2018-01-05 16:19:17,390 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-05 16:19:17,391 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-05 16:19:17,391 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-05 16:19:17,400 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-05 16:19:17,415 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-05 16:19:17,435 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/10.128.0.3:47007 2018-01-05 16:19:17,436 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:17,436 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:17,440 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:17,450 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:17,468 INFO [main] zookeeper.RecoverableZooKeeper(107): Process identifier=regionserver:47007 connecting to ZooKeeper ensemble=localhost:55510 2018-01-05 16:19:17,495 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:470070x0, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 16:19:17,502 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:47007-0x160c71e6b450001 connected 2018-01-05 16:19:17,503 DEBUG [main] zookeeper.ZKUtil(355): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-05 16:19:17,509 DEBUG [main] zookeeper.ZKUtil(357): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 16:19:17,509 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=47007 2018-01-05 16:19:17,556 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=47007 2018-01-05 16:19:17,612 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=47007 2018-01-05 16:19:17,696 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:17,703 DEBUG [instance-2:54757.masterManager] util.FSUtils(665): Created cluster ID file at hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/hbase.id with ID: e5d606d3-8100-44da-ad00-ef4085e50e48 2018-01-05 16:19:17,823 INFO [RS:0;instance-2:47007] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x6eb53c4e to localhost:55510, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 16:19:17,823 INFO [M:0;instance-2:54757] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x7e9bdcdb to localhost:55510, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 16:19:17,885 WARN [M:0;instance-2:54757] client.ConnectionImplementation(528): Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:132) at org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) at org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:775) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:806) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:821) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:932) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:546) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) ... 1 more 2018-01-05 16:19:17,885 WARN [RS:0;instance-2:47007] client.ConnectionImplementation(528): Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) at org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:132) at org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) at org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:775) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:806) at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:821) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:932) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) ... 1 more 2018-01-05 16:19:17,892 DEBUG [RS:0;instance-2:47007] client.ConnectionImplementation(532): clusterid came back null, using default default-cluster 2018-01-05 16:19:17,893 DEBUG [M:0;instance-2:54757] client.ConnectionImplementation(532): clusterid came back null, using default default-cluster 2018-01-05 16:19:17,903 INFO [instance-2:54757.masterManager] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-05 16:19:17,945 INFO [instance-2:54757.masterManager] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995 Table name == hbase:meta 2018-01-05 16:19:18,031 DEBUG [M:0;instance-2:54757] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1117d89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 16:19:18,038 DEBUG [RS:0;instance-2:47007] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a15637d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 16:19:18,127 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:18,149 DEBUG [instance-2:54757.masterManager] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-05 16:19:18,356 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/info 2018-01-05 16:19:18,601 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:18,633 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:18,745 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:18,855 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_barrier 2018-01-05 16:19:18,855 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:18,856 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:18,857 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:18,866 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_meta 2018-01-05 16:19:18,866 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:18,867 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:18,869 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:18,879 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_position 2018-01-05 16:19:18,880 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:18,881 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:18,881 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:18,893 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/table 2018-01-05 16:19:18,894 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:18,894 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:18,896 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:18,926 DEBUG [instance-2:54757.masterManager] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740 2018-01-05 16:19:18,986 DEBUG [instance-2:54757.masterManager] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-05 16:19:19,002 DEBUG [instance-2:54757.masterManager] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 16:19:19,003 INFO [instance-2:54757.masterManager] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-05 16:19:19,003 DEBUG [instance-2:54757.masterManager] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-05 16:19:19,003 DEBUG [instance-2:54757.masterManager] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-05 16:19:19,018 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-05 16:19:19,020 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-05 16:19:19,021 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-05 16:19:19,022 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-05 16:19:19,022 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-05 16:19:19,023 INFO [instance-2:54757.masterManager] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-05 16:19:19,144 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:19,150 DEBUG [instance-2:54757.masterManager] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-05 16:19:19,194 INFO [instance-2:54757.masterManager] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-05 16:19:19,301 INFO [instance-2:54757.masterManager] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-05 16:19:19,303 DEBUG [instance-2:54757.masterManager] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-05 16:19:19,471 INFO [instance-2:54757.masterManager] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-05 16:19:19,506 INFO [instance-2:54757.masterManager] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-01-05 16:19:19,544 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-05 16:19:19,547 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-05 16:19:19,559 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-05 16:19:19,561 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-05 16:19:19,754 INFO [instance-2:54757.masterManager] assignment.AssignmentManager(219): Starting assignment manager 2018-01-05 16:19:19,890 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-05 16:19:19,890 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-05 16:19:19,895 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(867): ClusterId : e5d606d3-8100-44da-ad00-ef4085e50e48 2018-01-05 16:19:19,896 INFO [instance-2:54757.masterManager] master.HMaster(728): Server active/primary master=instance-2.c.gcp-hbase.internal,54757,1515169150919, sessionid=0x160c71e6b450000, setting cluster-up flag (Was=false) 2018-01-05 16:19:19,896 INFO [M:0;instance-2:54757] regionserver.HRegionServer(867): ClusterId : e5d606d3-8100-44da-ad00-ef4085e50e48 2018-01-05 16:19:19,912 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-05 16:19:19,964 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-05 16:19:19,969 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-05 16:19:19,972 DEBUG [instance-2:54757.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/flush-table-proc/acquired already exists 2018-01-05 16:19:19,977 INFO [instance-2:54757.masterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-05 16:19:19,988 DEBUG [instance-2:54757.masterManager] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:19,991 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-05 16:19:19,991 DEBUG [RS:0;instance-2:47007] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d338e3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=instance-2.c.gcp-hbase.internal/10.128.0.3:0 2018-01-05 16:19:19,999 DEBUG [instance-2:54757.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/online-snapshot/acquired already exists 2018-01-05 16:19:20,001 INFO [instance-2:54757.masterManager] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-05 16:19:20,001 DEBUG [RS:0;instance-2:47007] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;instance-2:47007 2018-01-05 16:19:20,004 DEBUG [instance-2:54757.masterManager] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:19:20,074 INFO [RS:0;instance-2:47007] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-05 16:19:20,074 INFO [RS:0;instance-2:47007] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-05 16:19:20,089 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(2620): reportForDuty to master=instance-2.c.gcp-hbase.internal,54757,1515169150919 with port=47007, startcode=1515169157369 2018-01-05 16:19:20,235 INFO [instance-2:54757.masterManager] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-05 16:19:20,263 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-instance-2:54757, corePoolSize=5, maxPoolSize=5 2018-01-05 16:19:20,263 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-instance-2:54757, corePoolSize=5, maxPoolSize=5 2018-01-05 16:19:20,263 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-instance-2:54757, corePoolSize=5, maxPoolSize=5 2018-01-05 16:19:20,263 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-instance-2:54757, corePoolSize=5, maxPoolSize=5 2018-01-05 16:19:20,264 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-instance-2:54757, corePoolSize=10, maxPoolSize=10 2018-01-05 16:19:20,264 DEBUG [instance-2:54757.masterManager] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-instance-2:54757, corePoolSize=1, maxPoolSize=1 2018-01-05 16:19:20,302 DEBUG [RS:0;instance-2:47007] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:54757 2018-01-05 16:19:20,401 INFO [instance-2:54757.masterManager] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-05 16:19:20,417 INFO [instance-2:54757.masterManager] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-05 16:19:20,468 DEBUG [instance-2:54757.masterManager] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-05 16:19:20,471 INFO [instance-2:54757.masterManager] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-05 16:19:20,472 INFO [instance-2:54757.masterManager] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 55msec 2018-01-05 16:19:20,475 DEBUG [instance-2:54757.masterManager] wal.WALProcedureStore(409): No state logs to replay. 2018-01-05 16:19:20,475 DEBUG [instance-2:54757.masterManager] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-05 16:19:20,476 INFO [instance-2:54757.masterManager] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 4msec 2018-01-05 16:19:20,585 INFO [instance-2:54757.masterManager] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515169190585 2018-01-05 16:19:20,591 INFO [instance-2:54757.masterManager] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-05 16:19:20,678 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-05 16:19:20,680 INFO [instance-2:54757.masterManager] zookeeper.RecoverableZooKeeper(107): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:55510 2018-01-05 16:19:20,697 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(527): Auth successful for jenkins.hfs.0 (auth:SIMPLE) 2018-01-05 16:19:20,711 DEBUG [instance-2:54757.masterManager-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-05 16:19:20,712 DEBUG [instance-2:54757.masterManager-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160c71e6b450004 connected 2018-01-05 16:19:20,712 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-05 16:19:20,714 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-05 16:19:20,714 INFO [instance-2:54757.masterManager] cleaner.CleanerChore(113): Cleaner pool size is 1 2018-01-05 16:19:20,714 INFO [instance-2:54757.masterManager] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-05 16:19:20,730 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 51602 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:20,767 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-05 16:19:20,785 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-05 16:19:20,787 DEBUG [instance-2:54757.masterManager] cleaner.CleanerChore(160): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-05 16:19:20,802 DEBUG [instance-2:54757.masterManager] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[instance-2:54757.masterManager-HFileCleaner.large.0-1515169160801,5,main] 2018-01-05 16:19:20,804 DEBUG [instance-2:54757.masterManager] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[instance-2:54757.masterManager-HFileCleaner.small.0-1515169160803,5,main] 2018-01-05 16:19:20,865 INFO [instance-2:54757.masterManager] master.HMaster(824): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515169157243, completionTime=-1 2018-01-05 16:19:20,871 INFO [instance-2:54757.masterManager] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=1 server(s), max=1 server(s), timeout=4500ms, lastChange=0ms 2018-01-05 16:19:20,880 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(2640): Master is not running yet 2018-01-05 16:19:20,880 WARN [RS:0;instance-2:47007] regionserver.HRegionServer(951): reportForDuty failed; sleeping and then retrying. 2018-01-05 16:19:21,881 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(2620): reportForDuty to master=instance-2.c.gcp-hbase.internal,54757,1515169150919 with port=47007, startcode=1515169157369 2018-01-05 16:19:21,975 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.ServerManager(432): Registering server=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:21,998 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1495): Config from master: hbase.rootdir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995 2018-01-05 16:19:22,000 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1495): Config from master: fs.defaultFS=hdfs://localhost:44287 2018-01-05 16:19:22,000 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1495): Config from master: hbase.master.info.port=-1 2018-01-05 16:19:22,013 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-05 16:19:22,014 DEBUG [RS:0;instance-2:47007] zookeeper.ZKUtil(355): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,014 WARN [RS:0;instance-2:47007] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-05 16:19:22,015 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1801): logDir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,024 INFO [instance-2:54757.masterManager] master.ServerManager(891): Finished wait on RegionServer count=1; waited=1153ms, expected min=1 server(s), max=1 server(s), master is running 2018-01-05 16:19:22,024 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,106 INFO [instance-2:54757.masterManager] master.HMaster(3513): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-05 16:19:22,187 INFO [RS:0;instance-2:47007] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-05 16:19:22,256 INFO [RS:0;instance-2:47007] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x234ab7d2 to localhost:55510, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 16:19:22,271 DEBUG [RS:0;instance-2:47007] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8127499, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 16:19:22,271 DEBUG [RS:0;instance-2:47007] regionserver.Replication(152): ReplicationStatisticsThread 300 2018-01-05 16:19:22,288 INFO [RS:0;instance-2:47007] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-05 16:19:22,413 INFO [RS:0;instance-2:47007] regionserver.MemStoreFlusher(105): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-01-05 16:19:22,438 DEBUG [instance-2:54757.masterManager] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 16:19:22,461 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 16:19:22,462 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 16:19:22,469 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-05 16:19:22,469 INFO [RS:0;instance-2:47007] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-05 16:19:22,473 INFO [RS:0;instance-2:47007] regionserver.HRegionServer$CompactionChecker(1695): CompactionChecker runs every 1sec 2018-01-05 16:19:22,475 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-05 16:19:22,486 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-05 16:19:22,487 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-05 16:19:22,487 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-05 16:19:22,496 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-05 16:19:22,525 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(813): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-05 16:19:22,540 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-05 16:19:22,571 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-instance-2:47007, corePoolSize=3, maxPoolSize=3 2018-01-05 16:19:22,574 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-instance-2:47007, corePoolSize=1, maxPoolSize=1 2018-01-05 16:19:22,575 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-instance-2:47007, corePoolSize=3, maxPoolSize=3 2018-01-05 16:19:22,575 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-instance-2:47007, corePoolSize=3, maxPoolSize=3 2018-01-05 16:19:22,575 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-instance-2:47007, corePoolSize=1, maxPoolSize=1 2018-01-05 16:19:22,576 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-instance-2:47007, corePoolSize=2, maxPoolSize=2 2018-01-05 16:19:22,577 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-instance-2:47007, corePoolSize=10, maxPoolSize=10 2018-01-05 16:19:22,580 DEBUG [RS:0;instance-2:47007] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-instance-2:47007, corePoolSize=3, maxPoolSize=3 2018-01-05 16:19:22,647 INFO [SplitLogWorker-instance-2:47007] regionserver.SplitLogWorker(130): SplitLogWorker instance-2.c.gcp-hbase.internal,47007,1515169157369 starting 2018-01-05 16:19:22,656 INFO [RS:0;instance-2:47007] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-05 16:19:22,683 DEBUG [ReplicationExecutor-0] zookeeper.ZKUtil(355): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,684 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$AdoptAbandonedQueuesWorker(788): Current list of replicators: [instance-2.c.gcp-hbase.internal,47007,1515169157369] other RSs: [instance-2.c.gcp-hbase.internal,47007,1515169157369] 2018-01-05 16:19:22,694 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=1, allServersCount=1 2018-01-05 16:19:22,706 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,830 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1536): Serving as instance-2.c.gcp-hbase.internal,47007,1515169157369, RpcServer on instance-2.c.gcp-hbase.internal/10.128.0.3:47007, sessionid=0x160c71e6b450001 2018-01-05 16:19:22,831 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-05 16:19:22,831 DEBUG [RS:0;instance-2:47007] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,832 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'instance-2.c.gcp-hbase.internal,47007,1515169157369' 2018-01-05 16:19:22,832 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 16:19:22,837 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 16:19:22,840 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-05 16:19:22,840 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-05 16:19:22,841 DEBUG [RS:0;instance-2:47007] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:22,841 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'instance-2.c.gcp-hbase.internal,47007,1515169157369' 2018-01-05 16:19:22,841 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-05 16:19:22,849 DEBUG [RS:0;instance-2:47007] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-05 16:19:22,851 DEBUG [RS:0;instance-2:47007] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-05 16:19:22,855 INFO [RS:0;instance-2:47007] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-05 16:19:22,855 INFO [RS:0;instance-2:47007] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-05 16:19:23,979 INFO [RS:0;instance-2:47007] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369, suffix=, logDir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369, archiveDir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs 2018-01-05 16:19:24,104 DEBUG [RS:0;instance-2:47007] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(417): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-01-05 16:19:24,203 DEBUG [RS:0;instance-2:47007] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(490): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:488) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:552) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:628) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:504) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:69) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:44) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:55) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:244) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2123) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1315) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1196) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1008) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:119) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:745) 2018-01-05 16:19:24,209 DEBUG [RS:0;instance-2:47007] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(524): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-01-05 16:19:24,332 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:657) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:117) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:716) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:711) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:745) 2018-01-05 16:19:24,338 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:51821,DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368,DISK] 2018-01-05 16:19:24,525 INFO [RS:0;instance-2:47007] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369/instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369.1515169164048 2018-01-05 16:19:24,525 DEBUG [RS:0;instance-2:47007] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:51821,DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368,DISK]] 2018-01-05 16:19:24,629 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:24,653 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-05 16:19:24,656 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(213): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:24,811 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:25,001 DEBUG [RSProcedureDispatcher-pool4-t1] master.ServerManager(783): New admin connection to instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:25,017 DEBUG [RSProcedureDispatcher-pool4-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:47007 2018-01-05 16:19:25,026 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 16:19:25,027 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 34209 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:25,057 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=47007] regionserver.RSRpcServices(1906): Open hbase:meta,,1.1588230740 2018-01-05 16:19:25,065 INFO [RS_OPEN_META-instance-2:47007-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-05 16:19:25,080 INFO [RS_OPEN_META-instance-2:47007-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369.meta, suffix=.meta, logDir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369, archiveDir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs 2018-01-05 16:19:25,108 DEBUG [RS-EventLoopGroup-3-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:51821,DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368,DISK] 2018-01-05 16:19:25,141 INFO [RS_OPEN_META-instance-2:47007-0] wal.AbstractFSWAL(708): New WAL /user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369/instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369.meta.1515169165083.meta 2018-01-05 16:19:25,141 DEBUG [RS_OPEN_META-instance-2:47007-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:51821,DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368,DISK]] 2018-01-05 16:19:25,143 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-05 16:19:25,355 DEBUG [RS_OPEN_META-instance-2:47007-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-05 16:19:25,581 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-05 16:19:25,625 INFO [RS_OPEN_META-instance-2:47007-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-05 16:19:25,675 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-05 16:19:25,677 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-05 16:19:25,749 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/info 2018-01-05 16:19:25,751 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/info 2018-01-05 16:19:25,752 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:25,755 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:25,760 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:25,777 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_barrier 2018-01-05 16:19:25,777 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_barrier 2018-01-05 16:19:25,778 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:25,779 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:25,785 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:25,797 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_meta 2018-01-05 16:19:25,798 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_meta 2018-01-05 16:19:25,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:25,802 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:25,804 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:25,817 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_position 2018-01-05 16:19:25,818 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/rep_position 2018-01-05 16:19:25,821 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:25,822 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:25,830 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:25,843 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/table 2018-01-05 16:19:25,844 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/table 2018-01-05 16:19:25,844 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:25,845 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:25,845 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:25,861 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740 2018-01-05 16:19:25,866 DEBUG [RS_OPEN_META-instance-2:47007-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-05 16:19:25,888 DEBUG [RS_OPEN_META-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-05 16:19:25,890 INFO [RS_OPEN_META-instance-2:47007-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-05 16:19:26,151 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2205): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-05 16:19:26,260 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:26,275 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2229): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-05 16:19:26,279 DEBUG [RS_OPEN_META-instance-2:47007-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:26,674 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169166327}]},"ts":1515169166327} 2018-01-05 16:19:26,728 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.AssignmentManager(996): META REPORTED: rit=OPEN, location=instance-2.c.gcp-hbase.internal,47007,1515169157369, table=hbase:meta, region=1588230740 2018-01-05 16:19:26,972 DEBUG [hconnection-0x130495de-shared-pool6-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:47007 2018-01-05 16:19:26,986 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 16:19:26,987 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 34228 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:27,352 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-05 16:19:27,353 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:27,360 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-05 16:19:27,362 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.AssignmentManager(998): META REPORTED but no procedure found (complete?); set location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:27,371 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-05 16:19:27,377 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-05 16:19:27,378 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 4.8660sec 2018-01-05 16:19:27,391 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 5.2670sec 2018-01-05 16:19:27,393 INFO [instance-2:54757.masterManager] master.HMaster(3516): hbase:meta (default replica) deployed at=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:27,853 INFO [instance-2:54757.masterManager] master.MasterWalManager(189): Log folder hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/WALs/instance-2.c.gcp-hbase.internal,47007,1515169157369 belongs to an existing region server 2018-01-05 16:19:27,854 INFO [instance-2:54757.masterManager] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-05 16:19:27,881 INFO [instance-2:54757.masterManager] assignment.AssignmentManager(1212): Number of RegionServers=1 2018-01-05 16:19:27,882 INFO [instance-2:54757.masterManager] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515169172882 2018-01-05 16:19:27,882 INFO [instance-2:54757.masterManager] assignment.AssignmentManager(1219): Joined the cluster in 28msec, failover=true 2018-01-05 16:19:28,088 INFO [instance-2:54757.masterManager] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-05 16:19:28,112 INFO [instance-2:54757.masterManager] master.HMaster(1765): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-05 16:19:28,958 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:19:28,968 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 16:19:29,083 DEBUG [instance-2:54757.masterManager] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:29,089 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:29,111 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:29,181 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:29,194 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-05 16:19:29,239 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp Table name == hbase:namespace 2018-01-05 16:19:29,321 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:29,323 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:29,325 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a.: disabling compactions & flushes 2018-01-05 16:19:29,325 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:29,326 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:29,342 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:29,408 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 16:19:29,442 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 16:19:29,460 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:29,461 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169169461}]},"ts":1515169169461} 2018-01-05 16:19:29,505 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-05 16:19:29,506 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369}] 2018-01-05 16:19:29,516 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(813): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 hbase:namespace hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:29,526 INFO [ProcExecWrkr-8] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369; forceNewPlan=false, retain=false 2018-01-05 16:19:29,682 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:29,748 INFO [ProcExecWrkr-9] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x67da8ca4 to localhost:55510, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 16:19:29,808 DEBUG [ProcExecWrkr-9] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@337c1612, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 16:19:29,817 DEBUG [MultiHConnection-shared--pool7-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:47007 2018-01-05 16:19:29,831 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 16:19:29,832 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 34356 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:29,851 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(213): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:30,001 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:30,012 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=47007] regionserver.RSRpcServices(1906): Open hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:30,048 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] regionserver.HRegion(6949): Opening region: {ENCODED => dfbe99206cf3d7bcfdc7c007d15aea7a, NAME => 'hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a.', STARTKEY => '', ENDKEY => ''} 2018-01-05 16:19:30,052 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace dfbe99206cf3d7bcfdc7c007d15aea7a 2018-01-05 16:19:30,055 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:30,083 DEBUG [StoreOpener-dfbe99206cf3d7bcfdc7c007d15aea7a-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/info 2018-01-05 16:19:30,111 DEBUG [StoreOpener-dfbe99206cf3d7bcfdc7c007d15aea7a-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/info 2018-01-05 16:19:30,111 INFO [StoreOpener-dfbe99206cf3d7bcfdc7c007d15aea7a-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-05 16:19:30,111 INFO [StoreOpener-dfbe99206cf3d7bcfdc7c007d15aea7a-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:30,112 INFO [StoreOpener-dfbe99206cf3d7bcfdc7c007d15aea7a-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:30,130 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a 2018-01-05 16:19:30,157 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 16:19:30,172 INFO [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] regionserver.HRegion(960): Onlined dfbe99206cf3d7bcfdc7c007d15aea7a; next sequenceid=2 2018-01-05 16:19:30,194 INFO [PostOpenDeployTasks:dfbe99206cf3d7bcfdc7c007d15aea7a] regionserver.HRegionServer(2205): Post open deploy tasks for hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:30,198 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:30,199 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:30,203 DEBUG [PostOpenDeployTasks:dfbe99206cf3d7bcfdc7c007d15aea7a] regionserver.HRegionServer(2229): Finished post open deploy task for hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:19:30,208 DEBUG [RS_OPEN_PRIORITY_REGION-instance-2:47007-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:30,274 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:30,281 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-05 16:19:30,281 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=dfbe99206cf3d7bcfdc7c007d15aea7a, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 760msec 2018-01-05 16:19:30,281 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:30,283 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169170283}]},"ts":1515169170283} 2018-01-05 16:19:30,302 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-05 16:19:30,314 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:30,325 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 2.1920sec 2018-01-05 16:19:30,338 DEBUG [instance-2:54757.masterManager] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-05 16:19:30,340 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-05 16:19:30,663 DEBUG [instance-2:54757.masterManager] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,664 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,682 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,699 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,801 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,804 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-05 16:19:30,821 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:30,827 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 341msec 2018-01-05 16:19:30,956 DEBUG [instance-2:54757.masterManager] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:30,958 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:30,968 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:30,974 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:30,985 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:30,987 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-05 16:19:30,995 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:31,003 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 146msec 2018-01-05 16:19:31,040 DEBUG [instance-2:54757.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/namespace/default already exists 2018-01-05 16:19:31,043 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-05 16:19:31,047 DEBUG [instance-2:54757.masterManager] zookeeper.RecoverableZooKeeper(631): Node /hbase/namespace/hbase already exists 2018-01-05 16:19:31,048 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-05 16:19:31,054 INFO [instance-2:54757.masterManager] master.HMaster(906): Master has completed initialization 13.753sec 2018-01-05 16:19:31,088 INFO [instance-2:54757.masterManager] quotas.MasterQuotaManager(80): Quota support disabled 2018-01-05 16:19:31,089 INFO [instance-2:54757.masterManager] zookeeper.ZKWatcher(187): not a secure deployment, proceeding 2018-01-05 16:19:31,238 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 16:19:31,284 INFO [main] zookeeper.ReadOnlyZKClient(129): Start read only zookeeper connection 0x5114b7c7 to localhost:55510, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-05 16:19:31,338 DEBUG [main] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dd71b20, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-05 16:19:31,413 DEBUG [hconnection-0x5854a18-shared-pool11-t1] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:47007 2018-01-05 16:19:31,497 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 16:19:31,498 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 34383 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:31,542 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up 2018-01-05 16:19:31,779 INFO [main] hbase.ResourceChecker(148): before: client.TestMultiRespectsLimits#testBlockMultiLimits Thread=208, OpenFileDescriptor=423, MaxFileDescriptor=4096, SystemLoadAverage=616, ProcessCount=95, AvailableMemoryMB=1396 2018-01-05 16:19:31,886 DEBUG [main] ipc.NettyRpcConnection(255): Connecting to instance-2.c.gcp-hbase.internal/10.128.0.3:54757 2018-01-05 16:19:31,909 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(527): Auth successful for jenkins (auth:SIMPLE) 2018-01-05 16:19:31,909 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(557): Connection from 10.128.0.3 port: 43007 with version info: version: "3.0.0-SNAPSHOT" url: "git://instance-2/var/lib/jenkins/workspace/HBase-Flaky-Tests" revision: "032fdc53de7bc1c59b2a44dda65d9fdf38eb9e3c" user: "jenkins" date: "Fri Jan 5 15:44:10 UTC 2018" src_checksum: "f688224303a54bf421559c08928d5f9c" version_major: 3 version_minor: 0 2018-01-05 16:19:32,014 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.HMaster$3(1732): Client=jenkins//10.128.0.3 create 'testBlockMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-05 16:19:32,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,124 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,148 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 16:19:32,227 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:32,235 DEBUG [ProcExecWrkr-14] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp/data/default/testBlockMultiLimits/.tabledesc/.tableinfo.0000000001 2018-01-05 16:19:32,241 INFO [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(6772): creating HRegion testBlockMultiLimits HTD == 'testBlockMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp Table name == testBlockMultiLimits 2018-01-05 16:19:32,318 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:32,326 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(807): Instantiated testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,328 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1481): Closing testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.: disabling compactions & flushes 2018-01-05 16:19:32,328 DEBUG [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1520): Updates disabled for region testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,328 INFO [RegionOpenAndInitThread-testBlockMultiLimits-1] regionserver.HRegion(1633): Closed testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 16:19:32,342 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,343 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 16:19:32,365 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 16:19:32,370 DEBUG [ProcExecWrkr-14] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,371 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testBlockMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169172370}]},"ts":1515169172370} 2018-01-05 16:19:32,387 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1786): Updated table testBlockMultiLimits state to ENABLING in META 2018-01-05 16:19:32,387 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369}] 2018-01-05 16:19:32,394 INFO [ProcExecWrkr-14] procedure.MasterProcedureScheduler(813): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 testBlockMultiLimits testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,399 INFO [ProcExecWrkr-14] assignment.AssignProcedure(211): Start pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369; forceNewPlan=false, retain=false 2018-01-05 16:19:32,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 16:19:32,551 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,567 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-05 16:19:32,571 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(138): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-01-05 16:19:32,572 INFO [ProcExecWrkr-16] assignment.RegionTransitionProcedure(213): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,722 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:32,725 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=47007] regionserver.RSRpcServices(1906): Open testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,737 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(6949): Opening region: {ENCODED => 92d345dd5f3cd9ac65407644f8bc8c2c, NAME => 'testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.', STARTKEY => '', ENDKEY => ''} 2018-01-05 16:19:32,740 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testBlockMultiLimits 92d345dd5f3cd9ac65407644f8bc8c2c 2018-01-05 16:19:32,740 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(807): Instantiated testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,757 DEBUG [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/D 2018-01-05 16:19:32,757 DEBUG [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/D 2018-01-05 16:19:32,792 INFO [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 16:19:32,793 INFO [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 16:19:32,793 INFO [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:32,794 INFO [StoreOpener-92d345dd5f3cd9ac65407644f8bc8c2c-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:32,803 INFO [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,808 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c 2018-01-05 16:19:32,816 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testBlockMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 16:19:32,842 DEBUG [RS_OPEN_REGION-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 16:19:32,842 INFO [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(960): Onlined 92d345dd5f3cd9ac65407644f8bc8c2c; next sequenceid=2 2018-01-05 16:19:32,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 16:19:32,848 INFO [PostOpenDeployTasks:92d345dd5f3cd9ac65407644f8bc8c2c] regionserver.HRegionServer(2205): Post open deploy tasks for testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,851 INFO [ProcExecWrkr-1] assignment.RegionStateStore(201): pid=8 updating hbase:meta row=testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,852 DEBUG [PostOpenDeployTasks:92d345dd5f3cd9ac65407644f8bc8c2c] regionserver.HRegionServer(2229): Finished post open deploy task for testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:32,853 DEBUG [RS_OPEN_REGION-instance-2:47007-0] handler.OpenRegionHandler(128): Opened testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,864 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:32,869 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testBlockMultiLimits; resume parent processing. 2018-01-05 16:19:32,869 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,870 DEBUG [ProcExecWrkr-2] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testBlockMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169172869}]},"ts":1515169172869} 2018-01-05 16:19:32,871 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=testBlockMultiLimits, region=92d345dd5f3cd9ac65407644f8bc8c2c, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 471msec 2018-01-05 16:19:32,888 INFO [ProcExecWrkr-2] hbase.MetaTableAccessor(1786): Updated table testBlockMultiLimits state to ENABLED in META 2018-01-05 16:19:32,892 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=7, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testBlockMultiLimits 2018-01-05 16:19:32,897 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; CreateTableProcedure table=testBlockMultiLimits in 878msec 2018-01-05 16:19:33,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=7 2018-01-05 16:19:33,356 INFO [main] client.HBaseAdmin$TableFuture(3632): Operation: CREATE, Table Name: default:testBlockMultiLimits completed 2018-01-05 16:19:33,538 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(849): Client=jenkins//10.128.0.3 procedure request for: flush-table-proc 2018-01-05 16:19:33,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] procedure.ProcedureCoordinator(177): Submitting procedure testBlockMultiLimits 2018-01-05 16:19:33,613 INFO [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(196): Starting procedure 'testBlockMultiLimits' 2018-01-05 16:19:33,614 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2018-01-05 16:19:33,615 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(204): Procedure 'testBlockMultiLimits' starting 'acquire' 2018-01-05 16:19:33,615 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(247): Starting procedure 'testBlockMultiLimits', kicking off acquire phase on members. 2018-01-05 16:19:33,618 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:33,618 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(93): Creating acquire znode:/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 16:19:33,620 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(101): Watching for acquire node:/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:33,620 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-05 16:19:33,620 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(105): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2018-01-05 16:19:33,620 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 16:19:33,621 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(183): Found procedure znode: /hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 16:19:33,622 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:33,622 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2018-01-05 16:19:33,622 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:33,622 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(209): start proc data length is 4 2018-01-05 16:19:33,623 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(211): Found data for znode:/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 16:19:33,623 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(154): Launching subprocedure to flush regions for testBlockMultiLimits 2018-01-05 16:19:33,645 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testBlockMultiLimits 2018-01-05 16:19:33,646 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(157): Starting subprocedure 'testBlockMultiLimits' with timeout 60000ms 2018-01-05 16:19:33,646 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2018-01-05 16:19:33,649 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(165): Subprocedure 'testBlockMultiLimits' starting 'acquire' stage 2018-01-05 16:19:33,653 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2018-01-05 16:19:33,653 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(245): Waiting for local region flush to finish. 2018-01-05 16:19:33,654 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:33,654 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. started... 2018-01-05 16:19:33,659 INFO [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=2.50 KB 2018-01-05 16:19:33,722 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.store: D 2018-01-05 16:19:33,746 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-01-05 16:19:34,132 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:34,136 INFO [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=10, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/.tmp/D/75f3a78504e04e1d92a341e0a1715c75 2018-01-05 16:19:34,360 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/.tmp/D/75f3a78504e04e1d92a341e0a1715c75 as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/D/75f3a78504e04e1d92a341e0a1715c75 2018-01-05 16:19:34,399 INFO [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.HStore(1028): Added hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/D/75f3a78504e04e1d92a341e0a1715c75, entries=6, sequenceid=10, filesize=5.2 K 2018-01-05 16:19:34,418 INFO [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] regionserver.HRegion(2712): Finished memstore flush of ~2.50 KB/2556, currentsize=0 B/0 for region testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. in 758ms, sequenceid=10, compaction requested=false 2018-01-05 16:19:34,422 DEBUG [rs(instance-2.c.gcp-hbase.internal,47007,1515169157369)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:19:34,422 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1/1 local region flush tasks. 2018-01-05 16:19:34,423 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(258): Completed 1 local region flush tasks. 2018-01-05 16:19:34,423 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(291): cancelling 0 flush region tasks instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,424 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(167): Subprocedure 'testBlockMultiLimits' locally acquired 2018-01-05 16:19:34,424 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(240): Member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' joining acquired barrier for procedure (testBlockMultiLimits) in zk 2018-01-05 16:19:34,427 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,427 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(248): Watch for global barrier reached:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,427 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,428 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 16:19:34,428 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 16:19:34,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 16:19:34,429 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] zookeeper.ZKUtil(357): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,429 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(172): Subprocedure 'testBlockMultiLimits' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2018-01-05 16:19:34,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 16:19:34,430 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,430 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,431 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 16:19:34,432 DEBUG [main-EventThread] procedure.Procedure(298): member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' joining acquired barrier for procedure 'testBlockMultiLimits' on coordinator 2018-01-05 16:19:34,432 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@15751fd4[Count = 0] remaining members to acquire global barrier 2018-01-05 16:19:34,432 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(212): Procedure 'testBlockMultiLimits' starting 'in-barrier' execution. 2018-01-05 16:19:34,432 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(117): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,435 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,435 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(78): Received created event:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,435 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(129): Received reached global barrier:/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,436 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(186): Subprocedure 'testBlockMultiLimits' received 'reached' from coordinator. 2018-01-05 16:19:34,436 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(188): Subprocedure 'testBlockMultiLimits' locally completed 2018-01-05 16:19:34,436 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.ZKProcedureMemberRpcs(264): Marking procedure 'testBlockMultiLimits' completed for member 'instance-2.c.gcp-hbase.internal,47007,1515169157369' in zk 2018-01-05 16:19:34,437 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,437 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2018-01-05 16:19:34,440 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(193): Subprocedure 'testBlockMultiLimits' has notified controller of completion 2018-01-05 16:19:34,440 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2018-01-05 16:19:34,440 DEBUG [member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1] procedure.Subprocedure(218): Subprocedure 'testBlockMultiLimits' completed. 2018-01-05 16:19:34,440 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,440 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,440 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 16:19:34,440 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 16:19:34,441 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 16:19:34,442 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 16:19:34,442 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,443 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,443 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 16:19:34,444 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,444 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,445 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(221): Finished data from procedure 'testBlockMultiLimits' member 'instance-2.c.gcp-hbase.internal,47007,1515169157369': 2018-01-05 16:19:34,445 INFO [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(221): Procedure 'testBlockMultiLimits' execution completed 2018-01-05 16:19:34,445 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(230): Running finish phase. 2018-01-05 16:19:34,445 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2018-01-05 16:19:34,445 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureCoordinator(164): Attempting to clean out zk node for op:testBlockMultiLimits 2018-01-05 16:19:34,445 INFO [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] procedure.ZKProcedureUtil(286): Clearing all znodes for procedure testBlockMultiLimitsincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-05 16:19:34,446 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' released barrier for procedure'testBlockMultiLimits', counting down latch. Waiting for 0 more 2018-01-05 16:19:34,451 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,451 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(78): Received created event:/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,451 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,452 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,452 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(196): Node created: /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,452 DEBUG [main-EventThread] procedure.ZKProcedureUtil(245): Current zk system: 2018-01-05 16:19:34,452 DEBUG [main-EventThread] procedure.ZKProcedureUtil(247): |-/hbase/flush-table-proc 2018-01-05 16:19:34,453 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-05 16:19:34,453 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(108): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2018-01-05 16:19:34,453 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 16:19:34,454 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,454 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-abort 2018-01-05 16:19:34,455 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,455 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,456 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-acquired 2018-01-05 16:19:34,456 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,457 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] zookeeper.ZKUtil(355): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,457 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,458 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-reached 2018-01-05 16:19:34,458 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |----testBlockMultiLimits 2018-01-05 16:19:34,462 DEBUG [main-EventThread] procedure.ZKProcedureUtil(264): |-------instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,472 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2018-01-05 16:19:34,472 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(105): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2018-01-05 16:19:34,472 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-05 16:19:34,474 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2018-01-05 16:19:34,474 DEBUG [(instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2018-01-05 16:19:34,474 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(108): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2018-01-05 16:19:34,475 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-05 16:19:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] flush.MasterFlushTableProcedureManager(168): Done waiting - exec procedure flush-table-proc for 'testBlockMultiLimits' 2018-01-05 16:19:34,476 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] flush.MasterFlushTableProcedureManager(170): Master flush table procedure is successful! 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKUtil(614): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testBlockMultiLimits because node does not exist (not an error) 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testBlockMultiLimits 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testBlockMultiLimits 2018-01-05 16:19:34,477 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testBlockMultiLimits 2018-01-05 16:19:34,483 DEBUG [main] client.HBaseAdmin(2777): Waiting a max of 300000 ms for procedure 'flush-table-proc : testBlockMultiLimits'' to complete. (max 20000 ms per retry) 2018-01-05 16:19:34,484 DEBUG [main] client.HBaseAdmin(2786): (#1) Sleeping: 100ms while waiting for procedure completion. 2018-01-05 16:19:34,584 DEBUG [main] client.HBaseAdmin(2792): Getting current status of procedure from master... 2018-01-05 16:19:34,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1109): Checking to see if procedure from request:flush-table-proc is done 2018-01-05 16:19:34,646 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 16:19:34,942 INFO [main] hbase.ResourceChecker(172): after: client.TestMultiRespectsLimits#testBlockMultiLimits Thread=217 (was 208) Potentially hanging thread: hconnection-0x130495de-shared-pool6-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5854a18-metaLookup-shared--pool12-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_245626759_1 at /127.0.0.1:36524 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130495de-shared-pool6-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130495de-shared-pool6-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5854a18-shared-pool11-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (instance-2.c.gcp-hbase.internal,54757,1515169150919)-proc-coordinator-pool2-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130495de-shared-pool6-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MultiHConnection-shared--pool7-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MultiHConnection-shared--pool7-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'instance-2.c.gcp-hbase.internal,47007,1515169157369' subprocedure-pool1-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RSProcedureDispatcher-pool4-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Default-IPC-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5854a18-metaLookup-shared--pool12-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5854a18-shared-pool11-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130495de-shared-pool6-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_REGION-instance-2:47007-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=427 (was 423) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=590 (was 616), ProcessCount=97 (was 95) - ProcessCount LEAK? -, AvailableMemoryMB=1385 (was 1396) 2018-01-05 16:19:34,998 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 16:19:34,999 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 16:19:35,028 INFO [main] hbase.ResourceChecker(148): before: client.TestMultiRespectsLimits#testMultiLimits Thread=217, OpenFileDescriptor=427, MaxFileDescriptor=4096, SystemLoadAverage=590, ProcessCount=95, AvailableMemoryMB=1385 2018-01-05 16:19:35,035 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.HMaster$3(1732): Client=jenkins//10.128.0.3 create 'testMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-01-05 16:19:35,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] procedure2.ProcedureExecutor(869): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:35,145 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:35,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 16:19:35,158 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:35,251 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:35,255 WARN [DataStreamer for file /user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp/data/default/testMultiLimits/.tmp/.tableinfo.0000000001] hdfs.DFSOutputStream$DataStreamer(718): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1245) at java.lang.Thread.join(Thread.java:1319) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.closeResponder(DFSOutputStream.java:716) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.endBlock(DFSOutputStream.java:476) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:652) 2018-01-05 16:19:35,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 16:19:35,261 DEBUG [ProcExecWrkr-3] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp/data/default/testMultiLimits/.tabledesc/.tableinfo.0000000001 2018-01-05 16:19:35,291 INFO [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(6772): creating HRegion testMultiLimits HTD == 'testMultiLimits', {NAME => 'D', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/.tmp Table name == testMultiLimits 2018-01-05 16:19:35,419 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:35,425 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(807): Instantiated testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,433 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1481): Closing testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.: disabling compactions & flushes 2018-01-05 16:19:35,433 DEBUG [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,433 INFO [RegionOpenAndInitThread-testMultiLimits-1] regionserver.HRegion(1633): Closed testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,444 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:35,446 DEBUG [ProcExecWrkr-3] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-05 16:19:35,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 16:19:35,476 INFO [ProcExecWrkr-3] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-05 16:19:35,482 DEBUG [ProcExecWrkr-3] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:35,482 DEBUG [ProcExecWrkr-3] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169175482}]},"ts":1515169175482} 2018-01-05 16:19:35,510 INFO [ProcExecWrkr-3] hbase.MetaTableAccessor(1786): Updated table testMultiLimits state to ENABLING in META 2018-01-05 16:19:35,510 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369}] 2018-01-05 16:19:35,517 INFO [ProcExecWrkr-4] procedure.MasterProcedureScheduler(813): pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 testMultiLimits testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,520 INFO [ProcExecWrkr-4] assignment.AssignProcedure(211): Start pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369; forceNewPlan=false, retain=false 2018-01-05 16:19:35,673 INFO [ProcExecWrkr-5] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:35,699 INFO [ProcExecWrkr-5] assignment.RegionTransitionProcedure(213): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:35,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 16:19:35,849 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:35,861 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.RSRpcServices(1906): Open testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,875 DEBUG [RS_OPEN_REGION-instance-2:47007-1] regionserver.HRegion(6949): Opening region: {ENCODED => 33163c996cfcd8e1c3bee698bce19c90, NAME => 'testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.', STARTKEY => '', ENDKEY => ''} 2018-01-05 16:19:35,876 DEBUG [RS_OPEN_REGION-instance-2:47007-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:19:35,876 DEBUG [RS_OPEN_REGION-instance-2:47007-1] regionserver.HRegion(807): Instantiated testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,896 DEBUG [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D 2018-01-05 16:19:35,897 DEBUG [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D 2018-01-05 16:19:35,899 INFO [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 16:19:35,899 INFO [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 16:19:35,899 INFO [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:35,900 INFO [StoreOpener-33163c996cfcd8e1c3bee698bce19c90-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:35,902 INFO [RS_OPEN_REGION-instance-2:47007-1] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,905 DEBUG [RS_OPEN_REGION-instance-2:47007-1] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:19:35,914 DEBUG [RS_OPEN_REGION-instance-2:47007-1] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 16:19:35,934 DEBUG [RS_OPEN_REGION-instance-2:47007-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-05 16:19:35,935 INFO [RS_OPEN_REGION-instance-2:47007-1] regionserver.HRegion(960): Onlined 33163c996cfcd8e1c3bee698bce19c90; next sequenceid=2 2018-01-05 16:19:35,943 INFO [PostOpenDeployTasks:33163c996cfcd8e1c3bee698bce19c90] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:35,960 INFO [ProcExecWrkr-6] assignment.RegionStateStore(201): pid=10 updating hbase:meta row=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90., regionState=OPEN, openSeqNum=2, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:35,977 DEBUG [PostOpenDeployTasks:33163c996cfcd8e1c3bee698bce19c90] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:35,979 DEBUG [RS_OPEN_REGION-instance-2:47007-1] handler.OpenRegionHandler(128): Opened testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:36,001 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:36,013 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMultiLimits; resume parent processing. 2018-01-05 16:19:36,014 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=10, ppid=9, state=SUCCESS; AssignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 480msec 2018-01-05 16:19:36,013 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:36,014 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"testMultiLimits","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515169176014}]},"ts":1515169176014} 2018-01-05 16:19:36,059 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table testMultiLimits state to ENABLED in META 2018-01-05 16:19:36,063 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=testMultiLimits 2018-01-05 16:19:36,069 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1250): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testMultiLimits in 1.0290sec 2018-01-05 16:19:36,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.MasterRpcServices(1152): Checking to see if procedure is done pid=9 2018-01-05 16:19:36,276 INFO [main] client.HBaseAdmin$TableFuture(3632): Operation: CREATE, Table Name: default:testMultiLimits completed 2018-01-05 16:19:36,277 DEBUG [main] hbase.HBaseTestingUtility(3262): Waiting until all regions of table testMultiLimits get assigned. Timeout = 60000ms 2018-01-05 16:19:36,279 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 16:19:36,394 INFO [main] hbase.HBaseTestingUtility(3314): All regions for table testMultiLimits assigned to meta. Checking AM states. 2018-01-05 16:19:36,416 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 16:19:36,417 INFO [main] hbase.HBaseTestingUtility(3334): All regions for table testMultiLimits assigned. 2018-01-05 16:19:37,453 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.RSRpcServices$1(1125): Large batch operation detected (greater than 5000) (HBASE-18023). Requested Number of Rows: 17576 Client: jenkins//10.128.0.3 first region in multi=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:38,178 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.HRegion(8152): writing data to region testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. with WAL disabled. Data may be lost in the event of a crash. 2018-01-05 16:19:38,243 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 16:19:38,818 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] master.HMaster$2(1605): Client=jenkins//10.128.0.3 split testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:38,822 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.SplitTableRegionProcedure(141): splitKey isn't explicitly specified, will try to find a best split key from RS 2018-01-05 16:19:38,832 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=480.59 KB 2018-01-05 16:19:38,833 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.CompactingMemStore(205): FLUSHING TO DISK: region testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.store: D 2018-01-05 16:19:38,833 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.CompactionPipeline(128): Swapping pipeline suffix; before=1, new segement=null 2018-01-05 16:19:39,118 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:19:39,126 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=5, memsize=480.6 K, hasBloomFilter=false, into tmp file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/.tmp/D/8bf865339e3641f5a1bec575552f0f4b 2018-01-05 16:19:39,167 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/.tmp/D/8bf865339e3641f5a1bec575552f0f4b as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b 2018-01-05 16:19:39,183 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.HStore(1028): Added hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b, entries=17576, sequenceid=5, filesize=502.8 K 2018-01-05 16:19:39,186 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=47007] regionserver.HRegion(2712): Finished memstore flush of ~480.59 KB/492128, currentsize=0 B/0 for region testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. in 355ms, sequenceid=5, compaction requested=false 2018-01-05 16:19:39,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.SplitTableRegionProcedure(153): Splittable=true rit=OPEN, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:39,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] procedure2.ProcedureExecutor(869): Stored pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:39,319 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(813): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 testMultiLimits testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,320 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:39,331 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:39,332 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-05 16:19:39,340 DEBUG [ProcExecWrkr-8] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:39,344 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369}] 2018-01-05 16:19:39,406 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(813): pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369 testMultiLimits testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,407 INFO [ProcExecWrkr-9] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90., regionState=CLOSING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:39,419 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(213): Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=CLOSING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:39,570 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:39,602 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.RSRpcServices(1531): Close 33163c996cfcd8e1c3bee698bce19c90 without moving 2018-01-05 16:19:39,613 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] handler.CloseRegionHandler(92): Processing close of testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,615 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1481): Closing testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.: disabling compactions & flushes 2018-01-05 16:19:39,641 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,662 INFO [StoreCloserThread-testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.-1] regionserver.HStore(930): Closed D 2018-01-05 16:19:39,692 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2018-01-05 16:19:39,697 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1633): Closed testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=CLOSING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:39,718 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=12 updating hbase:meta row=testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90., regionState=CLOSED 2018-01-05 16:19:39,720 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] handler.CloseRegionHandler(125): Closed testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90. 2018-01-05 16:19:39,774 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=12, ppid=11, state=SUCCESS; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:39,779 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58; resume parent processing. 2018-01-05 16:19:39,780 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=12, ppid=11, state=SUCCESS; UnassignProcedure table=testMultiLimits, region=33163c996cfcd8e1c3bee698bce19c90, server=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 424msec 2018-01-05 16:19:39,780 DEBUG [ProcExecWrkr-10] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:39,834 INFO [ProcExecWrkr-10] assignment.SplitTableRegionProcedure(587): pid=11 splitting 1 storefiles, region=33163c996cfcd8e1c3bee698bce19c90, threads=1 2018-01-05 16:19:39,834 INFO [ProcExecWrkr-10] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:39,842 DEBUG [StoreFileSplitter-%1$d-pool13-t1] assignment.SplitTableRegionProcedure(667): pid=11 splitting started for store file: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b for region: 33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:19:39,999 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 23 2018-01-05 16:19:40,478 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:40,483 DEBUG [StoreFileSplitter-%1$d-pool13-t1] assignment.SplitTableRegionProcedure(678): pid=11 splitting complete for store file: hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b for region: 33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:19:40,483 DEBUG [ProcExecWrkr-10] assignment.SplitTableRegionProcedure(648): pid=11 split storefiles for region 33163c996cfcd8e1c3bee698bce19c90 Daughter A: 1 storefiles, Daughter B: 1 storefiles. 2018-01-05 16:19:40,516 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:40,595 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741843_1019{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 53 2018-01-05 16:19:41,010 DEBUG [ProcExecWrkr-10] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,015 DEBUG [ProcExecWrkr-10] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,016 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(2131): Put {"totalColumns":3,"row":"testMultiLimits,,1515169175034.33163c996cfcd8e1c3bee698bce19c90.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":1515169181015},{"qualifier":"splitA","vlen":52,"tag":[],"timestamp":1515169181015},{"qualifier":"splitB","vlen":52,"tag":[],"timestamp":1515169181015}]},"ts":1515169181015} 2018-01-05 16:19:41,016 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":1515169181015},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":1515169181015}]},"ts":1515169181015} 2018-01-05 16:19:41,016 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":1515169181015},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":1515169181015}]},"ts":1515169181015} 2018-01-05 16:19:41,433 DEBUG [ProcExecWrkr-10] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,438 DEBUG [ProcExecWrkr-10] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,438 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369}, {pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369}] 2018-01-05 16:19:41,444 INFO [ProcExecWrkr-10] procedure.MasterProcedureScheduler(813): pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 testMultiLimits testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:41,445 INFO [ProcExecWrkr-12] procedure.MasterProcedureScheduler(813): pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 testMultiLimits testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:41,466 INFO [ProcExecWrkr-10] assignment.AssignProcedure(211): Start pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369; forceNewPlan=false, retain=false 2018-01-05 16:19:41,486 INFO [ProcExecWrkr-12] assignment.AssignProcedure(211): Start pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OFFLINE, location=instance-2.c.gcp-hbase.internal,47007,1515169157369; forceNewPlan=false, retain=false 2018-01-05 16:19:41,618 INFO [ProcExecWrkr-15] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:41,619 INFO [ProcExecWrkr-14] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58., regionState=OPENING, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:41,646 INFO [ProcExecWrkr-15] assignment.RegionTransitionProcedure(213): Dispatch pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:41,648 INFO [ProcExecWrkr-14] assignment.RegionTransitionProcedure(213): Dispatch pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:41,699 DEBUG [hconnection-0x5854a18-shared-pool11-t49] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=0 of 16 failed; retrying after sleep of 100 because: No server address listed in hbase:meta for region testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. containing row aaa 2018-01-05 16:19:41,797 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(108): Using procedure batch rpc execution for serverName=instance-2.c.gcp-hbase.internal,47007,1515169157369 version=3145728 2018-01-05 16:19:41,823 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.RSRpcServices(1906): Open testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:41,836 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] regionserver.RSRpcServices(1906): Open testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:41,836 DEBUG [hconnection-0x5854a18-shared-pool11-t49] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=1 of 16 failed; retrying after sleep of 201 because: No server address listed in hbase:meta for region testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. containing row aaa 2018-01-05 16:19:41,837 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(6949): Opening region: {ENCODED => 78b5ed3484ac9f1087e22d6128d17907, NAME => 'testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907.', STARTKEY => '', ENDKEY => 'njs'} 2018-01-05 16:19:41,837 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 78b5ed3484ac9f1087e22d6128d17907 2018-01-05 16:19:41,837 DEBUG [RS_OPEN_REGION-instance-2:47007-2] regionserver.HRegion(6949): Opening region: {ENCODED => 16e6e185370412500859ae03e613ea58, NAME => 'testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58.', STARTKEY => 'njs', ENDKEY => ''} 2018-01-05 16:19:41,838 DEBUG [RS_OPEN_REGION-instance-2:47007-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMultiLimits 16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,838 DEBUG [RS_OPEN_REGION-instance-2:47007-2] regionserver.HRegion(807): Instantiated testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:41,837 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(807): Instantiated testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:41,849 DEBUG [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D 2018-01-05 16:19:41,849 DEBUG [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D 2018-01-05 16:19:41,851 INFO [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 16:19:41,851 INFO [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 16:19:41,855 INFO [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:41,858 INFO [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:41,864 DEBUG [StoreOpener-16e6e185370412500859ae03e613ea58-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D 2018-01-05 16:19:41,866 DEBUG [StoreOpener-16e6e185370412500859ae03e613ea58-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D 2018-01-05 16:19:41,867 INFO [StoreOpener-16e6e185370412500859ae03e613ea58-1] regionserver.CompactingMemStore(147): Setting in-memory flush size threshold to 13421772 and immutable segments index to be of type CHUNK_MAP 2018-01-05 16:19:41,867 INFO [StoreOpener-16e6e185370412500859ae03e613ea58-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2018-01-05 16:19:41,867 INFO [StoreOpener-16e6e185370412500859ae03e613ea58-1] hfile.CacheConfig(239): Created cacheConfig for D: blockCache=LruBlockCache{blockCount=2, currentSize=767344, freeSize=1043194960, maxSize=1043962304, heapSize=767344, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-05 16:19:41,870 INFO [StoreOpener-16e6e185370412500859ae03e613ea58-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-05 16:19:41,956 DEBUG [StoreOpener-16e6e185370412500859ae03e613ea58-1] regionserver.HStore(549): loaded hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-top 2018-01-05 16:19:41,957 INFO [RS_OPEN_REGION-instance-2:47007-2] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:41,962 DEBUG [RS_OPEN_REGION-instance-2:47007-2] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58 2018-01-05 16:19:41,966 DEBUG [RS_OPEN_REGION-instance-2:47007-2] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 16:19:41,982 DEBUG [RS_OPEN_REGION-instance-2:47007-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/recovered.edits/7.seqid, newSeqId=7, maxSeqId=0 2018-01-05 16:19:41,983 INFO [RS_OPEN_REGION-instance-2:47007-2] regionserver.HRegion(960): Onlined 16e6e185370412500859ae03e613ea58; next sequenceid=7 2018-01-05 16:19:42,015 INFO [PostOpenDeployTasks:16e6e185370412500859ae03e613ea58] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:42,027 DEBUG [StoreOpener-78b5ed3484ac9f1087e22d6128d17907-1] regionserver.HStore(549): loaded hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-bottom 2018-01-05 16:19:42,028 INFO [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(1033): Setting FlushNonSloppyStoresFirstPolicy for the region=testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:42,031 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907 2018-01-05 16:19:42,038 DEBUG [RS_OPEN_REGION-instance-2:47007-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testMultiLimits descriptor;using region.getMemStoreFlushSize/# of families (128.0M)) instead. 2018-01-05 16:19:42,066 DEBUG [RS_OPEN_REGION-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/recovered.edits/6.seqid, newSeqId=6, maxSeqId=0 2018-01-05 16:19:42,066 INFO [RS_OPEN_REGION-instance-2:47007-0] regionserver.HRegion(960): Onlined 78b5ed3484ac9f1087e22d6128d17907; next sequenceid=6 2018-01-05 16:19:42,071 DEBUG [PostOpenDeployTasks:16e6e185370412500859ae03e613ea58] regionserver.CompactSplit(350): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:0), split_queue=0 2018-01-05 16:19:42,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=7, pid=14, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,075 INFO [ProcExecWrkr-16] assignment.RegionStateStore(201): pid=14 updating hbase:meta row=testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58., regionState=OPEN, openSeqNum=7, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,075 DEBUG [PostOpenDeployTasks:16e6e185370412500859ae03e613ea58] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:42,085 INFO [PostOpenDeployTasks:78b5ed3484ac9f1087e22d6128d17907] regionserver.HRegionServer(2205): Post open deploy tasks for testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:42,091 DEBUG [RS_OPEN_REGION-instance-2:47007-2] handler.OpenRegionHandler(128): Opened testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,091 DEBUG [PostOpenDeployTasks:78b5ed3484ac9f1087e22d6128d17907] regionserver.CompactSplit(350): Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0 2018-01-05 16:19:42,093 DEBUG [RS:0;instance-2:47007-shortCompactions-1515169182041] compactions.SortedCompactionPolicy(68): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2018-01-05 16:19:42,094 DEBUG [RS:0;instance-2:47007-longCompactions-1515169162433] compactions.SortedCompactionPolicy(68): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2018-01-05 16:19:42,098 DEBUG [hconnection-0x5854a18-shared-pool11-t49] client.ConnectionImplementation(900): locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=2 of 16 failed; retrying after sleep of 301 because: No server address listed in hbase:meta for region testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. containing row aaa 2018-01-05 16:19:42,114 DEBUG [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.HStore(1722): 16e6e185370412500859ae03e613ea58 - D: Initiating minor compaction (all files) 2018-01-05 16:19:42,115 INFO [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.HRegion(2066): Starting compaction on D in region testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:19:42,115 INFO [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.HStore(1356): Starting compaction of [hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-top] into tmpdir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/.tmp, totalSize=502.8 K 2018-01-05 16:19:42,124 DEBUG [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.HStore(1722): 78b5ed3484ac9f1087e22d6128d17907 - D: Initiating minor compaction (all files) 2018-01-05 16:19:42,126 INFO [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.HRegion(2066): Starting compaction on D in region testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:42,127 INFO [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.HStore(1356): Starting compaction of [hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-bottom] into tmpdir=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/.tmp, totalSize=502.8 K 2018-01-05 16:19:42,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54757] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=6, pid=13, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369; rit=OPENING, location=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,130 INFO [ProcExecWrkr-1] assignment.RegionStateStore(201): pid=13 updating hbase:meta row=testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., regionState=OPEN, openSeqNum=6, regionLocation=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,128 DEBUG [RS:0;instance-2:47007-shortCompactions-1515169182041] compactions.Compactor(204): Compacting hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-top, keycount=8788, bloomtype=NONE, size=502.8 K, encoding=NONE, seqNum=6, earliestPutTs=1515169178136 2018-01-05 16:19:42,130 DEBUG [PostOpenDeployTasks:78b5ed3484ac9f1087e22d6128d17907] regionserver.HRegionServer(2229): Finished post open deploy task for testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:19:42,144 DEBUG [RS:0;instance-2:47007-longCompactions-1515169162433] compactions.Compactor(204): Compacting hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-bottom, keycount=8788, bloomtype=NONE, size=502.8 K, encoding=NONE, seqNum=5, earliestPutTs=1515169178136 2018-01-05 16:19:42,150 DEBUG [RS_OPEN_REGION-instance-2:47007-0] handler.OpenRegionHandler(128): Opened testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. on instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,169 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=14, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,169 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=13, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:19:42,181 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=14, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=16e6e185370412500859ae03e613ea58, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 696msec 2018-01-05 16:19:42,186 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58; resume parent processing. 2018-01-05 16:19:42,186 DEBUG [ProcExecWrkr-2] procedure2.StateMachineProcedure(180): pid=11, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 2018-01-05 16:19:42,186 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1250): Finished pid=13, ppid=11, state=SUCCESS; AssignProcedure table=testMultiLimits, region=78b5ed3484ac9f1087e22d6128d17907, target=instance-2.c.gcp-hbase.internal,47007,1515169157369 in 726msec 2018-01-05 16:19:42,192 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1250): Finished pid=11, state=SUCCESS; SplitTableRegionProcedure table=testMultiLimits, parent=33163c996cfcd8e1c3bee698bce19c90, daughterA=78b5ed3484ac9f1087e22d6128d17907, daughterB=16e6e185370412500859ae03e613ea58 in 2.9910sec 2018-01-05 16:19:42,625 INFO [RS:0;instance-2:47007-shortCompactions-1515169182041] throttle.PressureAwareThroughputController(153): 16e6e185370412500859ae03e613ea58#D#compaction#2 average throughput is 0.54 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 10.00 MB/sec 2018-01-05 16:19:42,635 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:19:42,665 DEBUG [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/.tmp/D/6e238bb021b1491fa83f01625406cc8d as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/6e238bb021b1491fa83f01625406cc8d 2018-01-05 16:19:42,703 INFO [RS:0;instance-2:47007-longCompactions-1515169162433] throttle.PressureAwareThroughputController(153): 78b5ed3484ac9f1087e22d6128d17907#D#compaction#3 average throughput is 0.49 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 10.00 MB/sec 2018-01-05 16:19:42,767 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741845_1021{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 267035 2018-01-05 16:19:42,788 INFO [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.HStore(1486): Completed compaction of 1 (all) file(s) in D of 16e6e185370412500859ae03e613ea58 into 6e238bb021b1491fa83f01625406cc8d(size=246.5 K), total size for store is 246.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2018-01-05 16:19:42,790 INFO [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.CompactSplit$CompactionRunner(594): Completed compaction: regionName = testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58., storeName = D, priority = 15, time = 1515169182041; duration=0sec 2018-01-05 16:19:42,800 DEBUG [RS:0;instance-2:47007-shortCompactions-1515169182041] regionserver.CompactSplit$CompactionRunner(622): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0 2018-01-05 16:19:43,184 DEBUG [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/.tmp/D/745e09bd29f344c1966022b693aee300 as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/745e09bd29f344c1966022b693aee300 2018-01-05 16:19:43,251 INFO [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.HStore(1486): Completed compaction of 1 (all) file(s) in D of 78b5ed3484ac9f1087e22d6128d17907 into 745e09bd29f344c1966022b693aee300(size=260.8 K), total size for store is 260.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2018-01-05 16:19:43,252 INFO [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.CompactSplit$CompactionRunner(594): Completed compaction: regionName = testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., storeName = D, priority = 15, time = 1515169182086; duration=1sec 2018-01-05 16:19:43,252 DEBUG [RS:0;instance-2:47007-longCompactions-1515169162433] regionserver.CompactSplit$CompactionRunner(622): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0 2018-01-05 16:19:45,198 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-05 16:19:47,912 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-01-05 16:19:47,920 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:19:47,922 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:19:47,929 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:19:47,936 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 1940ms GC pool 'PS MarkSweep' had collection(s): count=1 time=1844ms GC pool 'PS Scavenge' had collection(s): count=1 time=183ms 2018-01-05 16:19:50,586 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-05 16:19:50,587 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-05 16:19:50,587 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-05 16:19:54,604 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:19:54,605 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:19:54,605 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:19:54,606 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 2933ms GC pool 'PS MarkSweep' had collection(s): count=1 time=2791ms GC pool 'PS Scavenge' had collection(s): count=1 time=401ms 2018-01-05 16:19:54,606 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 401 actions to finish on table: testMultiLimits 2018-01-05 16:20:11,374 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 9557ms GC pool 'PS MarkSweep' had collection(s): count=1 time=8882ms GC pool 'PS Scavenge' had collection(s): count=1 time=1135ms 2018-01-05 16:20:11,378 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:20:11,378 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:20:11,378 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 346 actions to finish on table: testMultiLimits 2018-01-05 16:20:11,387 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:20:11,421 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] ipc.RpcServer(505): (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1515169201342,"responsesize":266044,"method":"Multi","param":"region= testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., for 1 actions and 1st row key=afy","processingtimems":10025,"client":"10.128.0.3:34383","queuetimems":0,"class":"MiniHBaseClusterRegionServer"} 2018-01-05 16:20:16,510 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:20:16,517 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 1140ms GC pool 'PS Scavenge' had collection(s): count=1 time=1444ms 2018-01-05 16:20:16,522 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:20:16,522 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:20:17,885 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x6eb53c4e no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:20:20,815 DEBUG [ForkJoinPool-1-worker-1] cleaner.CleanerChore$CleanerTask(403): CleanerTask 476 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive and itself. 2018-01-05 16:20:20,865 DEBUG [instance-2.c.gcp-hbase.internal,54757,1515169150919_Chore_1] cleaner.CleanerChore(217): Cleaned old files/dirs under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive successfully 2018-01-05 16:20:20,867 DEBUG [ForkJoinPool-1-worker-1] cleaner.CleanerChore$CleanerTask(403): CleanerTask 476 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs and itself. 2018-01-05 16:20:20,869 DEBUG [instance-2.c.gcp-hbase.internal,54757,1515169150919_Chore_1] cleaner.CleanerChore(217): Cleaned old files/dirs under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs successfully 2018-01-05 16:20:34,166 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x67da8ca4 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:20:34,166 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x7e9bdcdb no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:20:34,178 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@2a492f2a] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 12656ms GC pool 'PS MarkSweep' had collection(s): count=1 time=12151ms GC pool 'PS Scavenge' had collection(s): count=1 time=866ms 2018-01-05 16:20:34,172 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_3] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:20:34,172 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 280 actions to finish on table: testMultiLimits 2018-01-05 16:20:34,171 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:20:34,171 WARN [M:0;instance-2:54757] util.Sleeper(97): We slept 13659ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:20:34,171 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@a8a8b75] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 12656ms GC pool 'PS MarkSweep' had collection(s): count=1 time=12151ms GC pool 'PS Scavenge' had collection(s): count=1 time=866ms 2018-01-05 16:20:34,170 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_1] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:20:34,169 WARN [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(176): Detected pause in JVM or host machine (eg GC): pause of approximately 12622ms GC pool 'PS MarkSweep' had collection(s): count=1 time=12151ms GC pool 'PS Scavenge' had collection(s): count=1 time=866ms 2018-01-05 16:20:34,169 WARN [RS:0;instance-2:47007] util.Sleeper(97): We slept 13617ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:20:34,168 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x234ab7d2 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:20:34,167 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x5114b7c7 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:20:34,208 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=47007] ipc.RpcServer(505): (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1515169221142,"responsesize":215158,"method":"Multi","param":"region= testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., for 1 actions and 1st row key=aim","processingtimems":13066,"client":"10.128.0.3:34383","queuetimems":1,"class":"MiniHBaseClusterRegionServer"} 2018-01-05 16:20:59,818 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:20:59,818 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:20:59,819 WARN [RS:0;instance-2:47007] util.Sleeper(97): We slept 20593ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:20:59,820 WARN [M:0;instance-2:54757] util.Sleeper(97): We slept 20637ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:20:59,830 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:20:59,842 WARN [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(176): Detected pause in JVM or host machine (eg GC): pause of approximately 20157ms GC pool 'PS MarkSweep' had collection(s): count=1 time=18135ms GC pool 'PS Scavenge' had collection(s): count=1 time=2078ms 2018-01-05 16:20:59,843 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@2a492f2a] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 20158ms GC pool 'PS MarkSweep' had collection(s): count=1 time=18135ms GC pool 'PS Scavenge' had collection(s): count=1 time=2078ms 2018-01-05 16:20:59,867 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 238 actions to finish on table: testMultiLimits 2018-01-05 16:20:59,869 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@a8a8b75] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 20158ms GC pool 'PS MarkSweep' had collection(s): count=1 time=18135ms GC pool 'PS Scavenge' had collection(s): count=1 time=2078ms 2018-01-05 16:21:06,345 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:21:06,348 INFO [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(178): Detected pause in JVM or host machine (eg GC): pause of approximately 1003ms GC pool 'PS Scavenge' had collection(s): count=1 time=1168ms 2018-01-05 16:21:09,854 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 159 actions to finish on table: testMultiLimits 2018-01-05 16:21:11,325 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:21:11,325 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:21:17,898 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x6eb53c4e no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:21:33,931 WARN [RS:0;instance-2:47007] util.Sleeper(97): We slept 15516ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:21:33,933 WARN [M:0;instance-2:54757] util.Sleeper(97): We slept 15578ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2018-01-05 16:21:33,934 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: MemstoreFlusherChore missed its start time 2018-01-05 16:21:33,934 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(176): Chore: CompactionChecker missed its start time 2018-01-05 16:21:33,934 INFO [instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager__Chore_1] hbase.ScheduledChore(176): Chore: SplitLogManager Timeout Monitor missed its start time 2018-01-05 16:21:33,977 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@a8a8b75] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 15120ms GC pool 'PS MarkSweep' had collection(s): count=1 time=14327ms GC pool 'PS Scavenge' had collection(s): count=1 time=1124ms 2018-01-05 16:21:33,977 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@2a492f2a] util.JvmPauseMonitor$Monitor(191): Detected pause in JVM or host machine (eg GC): pause of approximately 15122ms GC pool 'PS MarkSweep' had collection(s): count=1 time=14327ms GC pool 'PS Scavenge' had collection(s): count=1 time=1124ms 2018-01-05 16:21:33,977 WARN [JvmPauseMonitor] util.JvmPauseMonitor$Monitor(176): Detected pause in JVM or host machine (eg GC): pause of approximately 15122ms GC pool 'PS MarkSweep' had collection(s): count=1 time=14327ms GC pool 'PS Scavenge' had collection(s): count=1 time=1124ms 2018-01-05 16:21:34,040 INFO [main] client.AsyncRequestFutureImpl(1221): #5, waiting for 66 actions to finish on table: testMultiLimits 2018-01-05 16:21:34,121 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs and itself. 2018-01-05 16:21:34,122 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-8] regionserver.HStore(2521): Moving the files [hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-top] to archive 2018-01-05 16:21:34,125 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-1] regionserver.HStore(2521): Moving the files [hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90-hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/33163c996cfcd8e1c3bee698bce19c90/D/8bf865339e3641f5a1bec575552f0f4b-bottom] to archive 2018-01-05 16:21:34,186 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x67da8ca4 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:21:34,202 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x7e9bdcdb no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:21:34,203 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x5114b7c7 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:21:34,203 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x234ab7d2 no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2018-01-05 16:21:34,244 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-1] backup.HFileArchiver(258): Archiving compacted store files. 2018-01-05 16:21:34,246 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-8] backup.HFileArchiver(258): Archiving compacted store files. 2018-01-05 16:21:34,332 DEBUG [instance-2.c.gcp-hbase.internal,54757,1515169150919_Chore_1] cleaner.CleanerChore(217): Cleaned old files/dirs under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs successfully 2018-01-05 16:21:34,334 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive and itself. 2018-01-05 16:21:34,338 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data and itself. 2018-01-05 16:21:34,344 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default and itself. 2018-01-05 16:21:34,349 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits and itself. 2018-01-05 16:21:34,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-1] backup.HFileArchiver(450): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90, to hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:21:34,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-instance-2:47007-8] backup.HFileArchiver(450): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90, to hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D/8bf865339e3641f5a1bec575552f0f4b.33163c996cfcd8e1c3bee698bce19c90 2018-01-05 16:21:34,488 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58 and itself. 2018-01-05 16:21:34,498 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/D and itself. 2018-01-05 16:21:34,513 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907 and itself. 2018-01-05 16:21:34,515 DEBUG [ForkJoinPool-1-worker-0] cleaner.CleanerChore$CleanerTask(403): CleanerTask 492 starts cleaning dirs and files under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/D and itself. 2018-01-05 16:21:34,535 WARN [instance-2.c.gcp-hbase.internal,54757,1515169150919_Chore_1] cleaner.CleanerChore(219): Failed to fully clean old files/dirs under hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/archive 2018-01-05 16:21:36,127 INFO [main] hbase.ResourceChecker(172): after: client.TestMultiRespectsLimits#testMultiLimits Thread=206 (was 217), OpenFileDescriptor=407 (was 427), MaxFileDescriptor=4096 (was 4096), SystemLoadAverage=819 (was 590) - SystemLoadAverage LEAK? -, ProcessCount=96 (was 95) - ProcessCount LEAK? -, AvailableMemoryMB=843 (was 1385) 2018-01-05 16:21:36,128 INFO [main] hbase.HBaseTestingUtility(1085): Shutting down minicluster 2018-01-05 16:21:36,129 INFO [main] client.ConnectionImplementation(1763): Closing master protocol: MasterService 2018-01-05 16:21:36,129 INFO [main] zookeeper.ReadOnlyZKClient(332): Close zookeeper connection 0x5114b7c7 to localhost:55510 2018-01-05 16:21:36,130 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:36,143 DEBUG [main] util.JVMClusterUtil(237): Shutting down HBase Cluster 2018-01-05 16:21:36,144 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:21:36,148 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-05 16:21:36,148 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-05 16:21:36,149 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 16:21:36,149 INFO [main] regionserver.HRegionServer(2166): ***** STOPPING region server 'instance-2.c.gcp-hbase.internal,47007,1515169157369' ***** 2018-01-05 16:21:36,149 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-05 16:21:36,150 INFO [main] regionserver.HRegionServer(2180): STOPPED: Shutdown requested 2018-01-05 16:21:36,150 INFO [RS:0;instance-2:47007] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-05 16:21:36,151 INFO [SplitLogWorker-instance-2:47007] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-05 16:21:36,151 INFO [RS:0;instance-2:47007] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-05 16:21:36,151 INFO [RS:0;instance-2:47007] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-01-05 16:21:36,151 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-05 16:21:36,151 INFO [SplitLogWorker-instance-2:47007] regionserver.SplitLogWorker(151): SplitLogWorker instance-2.c.gcp-hbase.internal,47007,1515169157369 exiting 2018-01-05 16:21:36,151 INFO [RS:0;instance-2:47007] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-01-05 16:21:36,170 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-05 16:21:36,183 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1082): stopping server instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:21:36,183 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:21:36,183 DEBUG [RS:0;instance-2:47007] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-05 16:21:36,184 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a.: disabling compactions & flushes 2018-01-05 16:21:36,184 INFO [RS:0;instance-2:47007] zookeeper.ReadOnlyZKClient(332): Close zookeeper connection 0x6eb53c4e to localhost:55510 2018-01-05 16:21:36,184 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:21:36,185 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=78 B 2018-01-05 16:21:36,184 DEBUG [RS:0;instance-2:47007] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:36,185 DEBUG [RS_CLOSE_REGION-instance-2:47007-1] handler.CloseRegionHandler(92): Processing close of testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:21:36,185 DEBUG [RS_CLOSE_REGION-instance-2:47007-1] regionserver.HRegion(1481): Closing testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.: disabling compactions & flushes 2018-01-05 16:21:36,185 DEBUG [RS_CLOSE_REGION-instance-2:47007-1] regionserver.HRegion(1520): Updates disabled for region testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:21:36,208 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] handler.CloseRegionHandler(92): Processing close of testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:21:36,208 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1481): Closing testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907.: disabling compactions & flushes 2018-01-05 16:21:36,208 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:21:36,246 INFO [RS:0;instance-2:47007] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-05 16:21:36,246 INFO [RS:0;instance-2:47007] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-05 16:21:36,246 INFO [RS:0;instance-2:47007] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-05 16:21:36,251 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1389): Waiting on 5 regions to close 2018-01-05 16:21:36,251 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1393): Online Regions={92d345dd5f3cd9ac65407644f8bc8c2c=testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c., 78b5ed3484ac9f1087e22d6128d17907=testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907., 1588230740=hbase:meta,,1.1588230740, dfbe99206cf3d7bcfdc7c007d15aea7a=hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a., 16e6e185370412500859ae03e613ea58=testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58.} 2018-01-05 16:21:36,288 DEBUG [RS_CLOSE_META-instance-2:47007-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-05 16:21:36,338 DEBUG [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-05 16:21:36,339 DEBUG [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-05 16:21:36,339 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegion(2516): Flushing 5/5 column families, memstore=7.50 KB 2018-01-05 16:21:36,340 INFO [StoreCloserThread-testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c.-1] regionserver.HStore(930): Closed D 2018-01-05 16:21:36,340 INFO [StoreCloserThread-testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907.-1] regionserver.HStore(930): Closed D 2018-01-05 16:21:36,589 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/78b5ed3484ac9f1087e22d6128d17907/recovered.edits/9.seqid, newSeqId=9, maxSeqId=6 2018-01-05 16:21:36,595 DEBUG [RS_CLOSE_REGION-instance-2:47007-1] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testBlockMultiLimits/92d345dd5f3cd9ac65407644f8bc8c2c/recovered.edits/13.seqid, newSeqId=13, maxSeqId=2 2018-01-05 16:21:36,631 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|FINALIZED]]} size 0 2018-01-05 16:21:36,633 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=78, hasBloomFilter=true, into tmp file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/.tmp/info/eac920ad54ea4e0a855591353a520436 2018-01-05 16:21:36,639 INFO [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1633): Closed testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:21:36,640 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] handler.CloseRegionHandler(125): Closed testMultiLimits,,1515169179193.78b5ed3484ac9f1087e22d6128d17907. 2018-01-05 16:21:36,640 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] handler.CloseRegionHandler(92): Processing close of testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:21:36,640 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1481): Closing testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58.: disabling compactions & flushes 2018-01-05 16:21:36,640 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1520): Updates disabled for region testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:21:36,643 INFO [RS_CLOSE_REGION-instance-2:47007-1] regionserver.HRegion(1633): Closed testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:21:36,643 DEBUG [RS_CLOSE_REGION-instance-2:47007-1] handler.CloseRegionHandler(125): Closed testBlockMultiLimits,,1515169171978.92d345dd5f3cd9ac65407644f8bc8c2c. 2018-01-05 16:21:36,775 INFO [StoreCloserThread-testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58.-1] regionserver.HStore(930): Closed D 2018-01-05 16:21:36,848 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/.tmp/info/eac920ad54ea4e0a855591353a520436 as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/info/eac920ad54ea4e0a855591353a520436 2018-01-05 16:21:36,848 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:21:36,850 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=28, memsize=7.2 K, hasBloomFilter=false, into tmp file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/.tmp/info/511ef6bf49b04380b2efc49713ae6bae 2018-01-05 16:21:36,853 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/default/testMultiLimits/16e6e185370412500859ae03e613ea58/recovered.edits/10.seqid, newSeqId=10, maxSeqId=7 2018-01-05 16:21:36,857 INFO [RS_CLOSE_REGION-instance-2:47007-2] regionserver.HRegion(1633): Closed testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:21:36,857 DEBUG [RS_CLOSE_REGION-instance-2:47007-2] handler.CloseRegionHandler(125): Closed testMultiLimits,njs,1515169179193.16e6e185370412500859ae03e613ea58. 2018-01-05 16:21:36,913 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HStore(1028): Added hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/info/eac920ad54ea4e0a855591353a520436, entries=2, sequenceid=6, filesize=4.8 K 2018-01-05 16:21:36,915 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(2712): Finished memstore flush of ~78 B/78, currentsize=0 B/0 for region hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. in 731ms, sequenceid=6, compaction requested=false 2018-01-05 16:21:36,936 INFO [regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0.leaseChecker] regionserver.Leases(147): regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0.leaseChecker closing leases 2018-01-05 16:21:36,937 INFO [regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0.leaseChecker] regionserver.Leases(150): regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0.leaseChecker closed leases 2018-01-05 16:21:36,958 INFO [instance-2.c.gcp-hbase.internal,47007,1515169157369_Chore_2] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-05 16:21:37,100 INFO [StoreCloserThread-hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a.-1] regionserver.HStore(930): Closed info 2018-01-05 16:21:37,132 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741848_1024{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 4992 2018-01-05 16:21:37,178 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/namespace/dfbe99206cf3d7bcfdc7c007d15aea7a/recovered.edits/9.seqid, newSeqId=9, maxSeqId=2 2018-01-05 16:21:37,180 INFO [RS_CLOSE_REGION-instance-2:47007-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:21:37,181 DEBUG [RS_CLOSE_REGION-instance-2:47007-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515169168092.dfbe99206cf3d7bcfdc7c007d15aea7a. 2018-01-05 16:21:37,252 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1389): Waiting on 1 regions to close 2018-01-05 16:21:37,253 DEBUG [RS:0;instance-2:47007] regionserver.HRegionServer(1393): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-01-05 16:21:37,535 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=28, memsize=334, hasBloomFilter=false, into tmp file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/.tmp/table/fa0a8abfdd804f70ba976506da34d3b5 2018-01-05 16:21:37,561 DEBUG [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/.tmp/info/511ef6bf49b04380b2efc49713ae6bae as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/info/511ef6bf49b04380b2efc49713ae6bae 2018-01-05 16:21:37,614 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.HStore(1028): Added hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/info/511ef6bf49b04380b2efc49713ae6bae, entries=53, sequenceid=28, filesize=11.3 K 2018-01-05 16:21:37,637 DEBUG [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/.tmp/table/fa0a8abfdd804f70ba976506da34d3b5 as hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/table/fa0a8abfdd804f70ba976506da34d3b5 2018-01-05 16:21:37,724 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.HStore(1028): Added hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/table/fa0a8abfdd804f70ba976506da34d3b5, entries=7, sequenceid=28, filesize=4.9 K 2018-01-05 16:21:37,743 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegion(2712): Finished memstore flush of ~7.50 KB/7682, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 1404ms, sequenceid=28, compaction requested=false 2018-01-05 16:21:37,794 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-05 16:21:37,795 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-05 16:21:37,795 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-05 16:21:37,795 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-05 16:21:37,824 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-05 16:21:37,862 DEBUG [RS_CLOSE_META-instance-2:47007-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44287/user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/data/hbase/meta/1588230740/recovered.edits/31.seqid, newSeqId=31, maxSeqId=3 2018-01-05 16:21:37,863 DEBUG [RS_CLOSE_META-instance-2:47007-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-05 16:21:37,866 INFO [RS_CLOSE_META-instance-2:47007-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-05 16:21:37,866 DEBUG [RS_CLOSE_META-instance-2:47007-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-05 16:21:38,053 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1110): stopping server instance-2.c.gcp-hbase.internal,47007,1515169157369; all regions closed. 2018-01-05 16:21:38,083 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:21:38,105 DEBUG [RS:0;instance-2:47007] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs 2018-01-05 16:21:38,106 INFO [RS:0;instance-2:47007] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369.meta:.meta(num 1515169165083) 2018-01-05 16:21:38,138 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-72f91e4b-6265-4e5e-9bc9-6cc7d1016368:NORMAL:127.0.0.1:51821|RBW]]} size 0 2018-01-05 16:21:38,150 DEBUG [RS:0;instance-2:47007] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/jenkins/test-data/8c3d1671-24bc-4d79-96c3-55a550660995/oldWALs 2018-01-05 16:21:38,150 INFO [RS:0;instance-2:47007] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL instance-2.c.gcp-hbase.internal%2C47007%2C1515169157369:(num 1515169164048) 2018-01-05 16:21:38,150 DEBUG [RS:0;instance-2:47007] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:38,150 INFO [RS:0;instance-2:47007] regionserver.Leases(147): RS:0;instance-2:47007 closing leases 2018-01-05 16:21:38,150 INFO [RS:0;instance-2:47007] regionserver.Leases(150): RS:0;instance-2:47007 closed leases 2018-01-05 16:21:38,152 INFO [RS:0;instance-2:47007] hbase.ChoreService(327): Chore service for: instance-2.c.gcp-hbase.internal,47007,1515169157369 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region instance-2.c.gcp-hbase.internal,47007,1515169157369 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-05 16:21:38,152 INFO [regionserver/instance-2.c.gcp-hbase.internal/10.128.0.3:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-05 16:21:38,155 INFO [RS:0;instance-2:47007] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-05 16:21:38,236 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:21:38,236 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/instance-2.c.gcp-hbase.internal,47007,1515169157369 2018-01-05 16:21:38,236 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:47007-0x160c71e6b450001, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-05 16:21:38,236 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [instance-2.c.gcp-hbase.internal,47007,1515169157369] 2018-01-05 16:21:38,238 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; instance-2.c.gcp-hbase.internal,47007,1515169157369 expired; onlineServers=0 2018-01-05 16:21:38,238 INFO [main-EventThread] regionserver.HRegionServer(2166): ***** STOPPING region server 'instance-2.c.gcp-hbase.internal,54757,1515169150919' ***** 2018-01-05 16:21:38,238 INFO [main-EventThread] regionserver.HRegionServer(2180): STOPPED: Cluster shutdown set; onlineServer=0 2018-01-05 16:21:38,238 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-05 16:21:38,239 DEBUG [M:0;instance-2:54757] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48ffbe0f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=instance-2.c.gcp-hbase.internal/10.128.0.3:0 2018-01-05 16:21:38,239 INFO [M:0;instance-2:54757] regionserver.HRegionServer(1082): stopping server instance-2.c.gcp-hbase.internal,54757,1515169150919 2018-01-05 16:21:38,239 DEBUG [M:0;instance-2:54757] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-05 16:21:38,239 INFO [M:0;instance-2:54757] zookeeper.ReadOnlyZKClient(332): Close zookeeper connection 0x7e9bdcdb to localhost:55510 2018-01-05 16:21:38,239 DEBUG [M:0;instance-2:54757] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:38,239 INFO [M:0;instance-2:54757] regionserver.HRegionServer(1110): stopping server instance-2.c.gcp-hbase.internal,54757,1515169150919; all regions closed. 2018-01-05 16:21:38,240 DEBUG [M:0;instance-2:54757] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:38,240 INFO [M:0;instance-2:54757] hbase.ChoreService(327): Chore service for: instance-2.c.gcp-hbase.internal,54757,1515169150919 had [[ScheduledChore: Name: instance-2.c.gcp-hbase.internal,54757,1515169150919-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: instance-2.c.gcp-hbase.internal,54757,1515169150919-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: instance-2.c.gcp-hbase.internal,54757,1515169150919-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: instance-2.c.gcp-hbase.internal,54757,1515169150919-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: ReplicationZKNodeCleanerChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationMetaCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-instance-2:54757 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: instance-2.c.gcp-hbase.internal,54757,1515169150919-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS]] on shutdown 2018-01-05 16:21:38,241 INFO [M:0;instance-2:54757] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-01-05 16:21:38,241 INFO [M:0;instance-2:54757] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-01-05 16:21:38,241 DEBUG [M:0;instance-2:54757] master.HMaster(1174): Stopping service threads 2018-01-05 16:21:38,249 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1156): stopping server instance-2.c.gcp-hbase.internal,47007,1515169157369; zookeeper connection closed. 2018-01-05 16:21:38,252 INFO [RS:0;instance-2:47007] regionserver.HRegionServer(1158): RS:0;instance-2:47007 exiting 2018-01-05 16:21:38,253 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@591e6e59] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@591e6e59 2018-01-05 16:21:38,262 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-05 16:21:38,262 INFO [M:0;instance-2:54757] assignment.AssignmentManager(236): Stopping assignment manager 2018-01-05 16:21:38,263 INFO [M:0;instance-2:54757] zookeeper.ReadOnlyZKClient(332): Close zookeeper connection 0x67da8ca4 to localhost:55510 2018-01-05 16:21:38,263 DEBUG [M:0;instance-2:54757] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-05 16:21:38,264 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:54757-0x160c71e6b450000, quorum=localhost:55510, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-05 16:21:38,274 INFO [M:0;instance-2:54757] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-01-05 16:21:38,275 INFO [M:0;instance-2:54757] procedure2.ProcedureExecutor(572): Stopping 2018-01-05 16:21:38,275 DEBUG [ProcExecWrkr-3] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,275 INFO [M:0;instance-2:54757] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-01-05 16:21:38,275 DEBUG [ProcExecWrkr-4] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,275 INFO [main] util.JVMClusterUtil(322): Shutdown of 1 master(s) and 1 regionserver(s) complete 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-5] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-6] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-7] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-8] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-9] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-13] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-10] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-12] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-15] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-14] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-16] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,276 DEBUG [ProcExecWrkr-11] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,277 DEBUG [ProcExecWrkr-1] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,278 DEBUG [ProcExecWrkr-2] procedure2.ProcedureExecutor$WorkerThread(1754): Worker terminated. 2018-01-05 16:21:38,327 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51821 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d739aef8-699c-4dc0-8f0e-fadfec954349:NORMAL:127.0.0.1:51821|RBW]]} size 214 2018-01-05 16:21:38,333 INFO [M:0;instance-2:54757] hbase.ChoreService(327): Chore service for: instance-2.c.gcp-hbase.internal,54757,1515169150919_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-01-05 16:21:38,333 INFO [M:0;instance-2:54757] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2018-01-05 16:21:38,334 INFO [M:0;instance-2:54757] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-05 16:21:38,367 DEBUG [M:0;instance-2:54757] zookeeper.RecoverableZooKeeper(181): Node /hbase/rs/instance-2.c.gcp-hbase.internal,54757,1515169150919 already deleted, retry=false 2018-01-05 16:21:38,373 INFO [M:0;instance-2:54757] regionserver.HRegionServer(1156): stopping server instance-2.c.gcp-hbase.internal,54757,1515169150919; zookeeper connection closed. 2018-01-05 16:21:38,373 INFO [M:0;instance-2:54757] regionserver.HRegionServer(1158): M:0;instance-2:54757 exiting 2018-01-05 16:21:38,375 WARN [main] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-01-05 16:21:38,499 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-01-05 16:21:38,605 WARN [DataNode: [[[DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data1/, [DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44287] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-01-05 16:21:38,606 WARN [DataNode: [[[DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data1/, [DISK]file:/var/lib/jenkins/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/b471f115-e150-429d-8119-1723ec467057/cluster_1dfa00c6-cc00-4b82-8902-e6602015f9cf/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44287] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1311183700-10.128.0.3-1515169137287 (Datanode Uuid cefd6a7c-2c7e-4d01-80b0-3bce99012e46) service to localhost/127.0.0.1:44287 2018-01-05 16:21:39,067 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-01-05 16:21:39,087 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-05 16:21:39,117 INFO [main] zookeeper.MiniZooKeeperCluster(324): Shutdown MiniZK cluster with all ZK servers 2018-01-05 16:21:39,178 DEBUG [instance-2:54757.masterManager-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner-0x160c71e6b450004, quorum=localhost:55510, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-01-05 16:21:39,179 DEBUG [instance-2:54757.masterManager-EventThread] zookeeper.ZKWatcher(528): replicationLogCleaner-0x160c71e6b450004, quorum=localhost:55510, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-01-05 16:21:39,435 INFO [main] hbase.HBaseTestingUtility(1096): Minicluster is down 2018-01-05 16:21:39,564 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1757cd72 2018-01-05 16:21:39,565 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-01-05 16:21:39,600 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.